A.21  Estimation Theory

A.21.1  Probabilistic estimation theory

Estimation theory concerns estimating the values of parameters $𝜃$ based on measured data. An estimator attempts to approximate the unknown parameters using measurements that have a random component (e. g., due to noise). There are other approaches, but the one of interest to this text is the probabilistic estimation theory, which aims at finding a set $\Theta =\{{𝜃}_1,\dots ,{𝜃}_M\}$ of $M$ parameters. In this approach, the measured data set $\mathrm{\Xi }$ with $N$ elements is random and with a probability distribution $p(\mathrm{\Xi }|\Theta )$ that depends on $\Theta$. The parameters $𝜃$ themselves may have associated probability distributions. An estimator provides $\widehat{\Theta }=\{{\widehat{𝜃}}_i\}$, where the “hat” suggests the element ${\widehat{𝜃}}_i$ is an estimate of the true parameter ${𝜃}_i$. The value ${\widehat{𝜃}}_i$ is a statistics with distribution, variance, mean $𝔼[{\widehat{𝜃}}_i]$, etc.

The bias of an estimator is $𝔼[{\widehat{𝜃}}_i]-{𝜃}_i$ and an estimator is called unbiased if the bias is zero for all $M$ elements of $\widehat{\Theta }$. The estimator of the sample variance that uses $N$ as normalization factor instead of $N-1$ is a good example of a biased estimator [ urlBMbia]. There is a well-known tradeoff between bias and variance. In many applications it is possible to decrease the variance of an estimator at the expenses of allowing its bias to increase.

Some specific topics of probabilistic estimation theory are discussed in the sequel. But this theory is broad and some of the (many) concepts out of the scope of this text are:

• Cramér-Rao bound (CRB): lower bound on the variance of an estimator;
• Several estimators: Maximum likelihood (MLE), maximum a posteriori (MAP), generalized Bayes, etc.;
• Non-linear estimators such as artificial neural networks;
• Dynamic systems: Kalman filter, recursive Bayesian estimation, etc.

A.21.2  Minimum mean square error (MMSE) estimators

A popular estimator is the minimum mean square error (MMSE) estimator, which utilizes the error $e_i={𝜃}_i-{\widehat{𝜃}}_i,i=1,\dots ,M,$ between the estimated parameters and their actual values.

The measurements $\mathrm{\Xi }$ may be organized in several different forms. For example, in multivariate statistics, each element of $\mathrm{\Xi }$ can be an array. For simplicity, it is assumed here that $\mathrm{\Xi }$ and $\Theta$ are column vectors $\mathbf{\text{y}}$ and $\mathbf{\text{x}}$ with $N$ and $M$ complex-valued elements, respectively. The estimator $\widehat{\mathbf{\text{x}}}(\mathbf{\text{y}})$ is a function of the measurements $\mathbf{\text{y}}$ and the estimation error is $\mathbf{\text{e}}=\widehat{\mathbf{\text{x}}}-\mathbf{\text{x}}$ with elements $e_i,i=1,\dots ,M$. Because the vectors are random, the MSE is given by

which can be written in matrix notation as the expected value of the trace

where $R_{\mathit{ee}}=𝔼[\mathbf{\text{e}}{\mathbf{\text{e}}}^H]$ is the error autocorrelation matrix.

A.21.3  Orthogonality principle

AK-TODO Orthogonality principle