Bias of an estimator

From Wikipedia, the free encyclopedia - View original article

Jump to: navigation, search





Suppose we have a statistical model parameterized by θ giving rise to a probability distribution for observed data, P_\theta(x) = P(x\mid\theta), and a statistic θ^ which serves as an estimator of θ based on any observed data x. That is, we assume that our data follow some unknown distribution P_\theta(x) = P(x\mid\theta) (where \theta is a fixed constant that is part of this distribution, but is unknown), and then we construct some estimator \hat\theta that maps observed data to values that we hope are close to \theta. Then the bias of this estimator (relative to the parameter θ) is defined to be

 \operatorname{Bias}_\theta[\,\hat\theta\,] = \operatorname{E}_\theta[\,\hat{\theta}\,]-\theta = \operatorname{E}_\theta[\, \hat\theta - \theta \,],

where \operatorname{E}_\theta denotes expected value over the distribution P_\theta(x) = P_\theta(x\mid\theta), i.e. averaging over all possible observations x. The second equation follows since θ is measurable with respect to the conditional distribution P(x\mid\theta).


There are more general notions of bias and unbiasedness. What this article calls "bias" is called "mean-bias", to distinguish mean-bias from the other notions, with the notable ones being "median-unbiased" estimators. For more details, the general theory of unbiased estimators is briefly discussed near the end of this article.



Sample variance[edit]



\overline{X}=\frac{1}{n}\sum_{i=1}^nX_i, \qquad S^2=\frac{1}{n}\sum_{i=1}^n\left(X_i-\overline{X}\,\right)^2,

then S2 is a biased estimator of σ2, because

     \begin{align}     \operatorname{E}[S^2]         &= \operatorname{E}\left[ \frac{1}{n}\sum_{i=1}^n \left(X_i-\overline{X}\right)^2 \right]          = \operatorname{E}\bigg[ \frac{1}{n}\sum_{i=1}^n \big((X_i-\mu)-(\overline{X}-\mu)\big)^2 \bigg] \\[8pt]         &= \operatorname{E}\bigg[ \frac{1}{n}\sum_{i=1}^n (X_i-\mu)^2 -                                   2(\overline{X}-\mu)\frac{1}{n}\sum_{i=1}^n (X_i-\mu) +                                   (\overline{X}-\mu)^2 \bigg] \\[8pt]         &= \operatorname{E}\bigg[ \frac{1}{n}\sum_{i=1}^n (X_i-\mu)^2 - (\overline{X}-\mu)^2 \bigg]          = \sigma^2 - \operatorname{E}\left[ (\overline{X}-\mu)^2 \right] < \sigma^2.     \end{align}

In other words, the expected value of the uncorrected sample variance does not equal the population variance σ2, unless multiplied by a normalization factor. The sample mean, on the other hand, is an unbiased estimator of the population mean μ.

The reason that S2 is biased stems from the fact that the sample mean is an ordinary least squares (OLS) estimator for μ: \overline{X} is the number that makes the sum \sum_{i=1}^N (X_i-\overline{X})^2 as small as possible. That is, when any other number is plugged into this sum, the sum can only increase. In particular, the choice \mu \ne \overline{X} gives,

         \frac{1}{n}\sum_{i=1}^n (X_i-\overline{X})^2  <  \frac{1}{n}\sum_{i=1}^n (X_i-\mu)^2,

and then

     \begin{align}     \operatorname{E}[S^2]         &= \operatorname{E}\bigg[ \frac{1}{n}\sum_{i=1}^n (X_i-\overline{X})^2 \bigg]          < \operatorname{E}\bigg[ \frac{1}{n}\sum_{i=1}^n (X_i-\mu)^2 \bigg]  = \sigma^2.     \end{align}

Note that the usual definition of sample variance is



\operatorname{E}\big[ (\overline{X}-\mu)^2 \big] = \frac{1}{n}\sigma^2 .


Estimating a Poisson probability[edit]

A far more extreme case of a biased estimator being better than any unbiased estimator arises from the Poisson distribution.[1][2] Suppose that X has a Poisson distribution with expectation λ. Suppose it is desired to estimate




E(\delta(X))=\sum_{x=0}^\infty \delta(x) \frac{\lambda^x e^{-\lambda}}{x!}=e^{-2\lambda},

the only function of the data constituting an unbiased estimator is

\delta(x)=(-1)^x. \,



The (biased) maximum likelihood estimator



e^{-4\lambda}-2e^{\lambda(1/e^2-3)}+e^{\lambda(1/e^4-1)} \,

is smaller; compare the unbiased estimator's MSE of

1-e^{-4\lambda}. \,


e^{-2\lambda}-e^{\lambda(1/e^2-1)}. \,

Maximum of a discrete uniform distribution[edit]


Median-unbiased estimators[edit]

The theory of median-unbiased estimators was revived by George W. Brown in 1947:[3]


Further properties of median-unbiased estimators have been noted by Lehmann, Birnbaum, van der Vaart and Pfanzagl.[citation needed] In particular, median-unbiased estimators exist in cases where mean-unbiased and maximum-likelihood estimators do not exist. Besides being invariant under one-to-one transformations, median-unbiased estimators have surprising robustness.[citation needed]

Bias with respect to other loss functions[edit]

Any minimum-variance mean-unbiased estimator minimizes the risk (expected loss) with respect to the squared-error loss function (among mean-unbiased estimators), as observed by Gauss.[citation needed] A minimum-average absolute deviation median-unbiased estimator minimizes the risk with respect to the absolute loss function (among median-unbiased estimators), as observed by Laplace.[citation needed] Other loss functions are used in statistical theory, particularly in robust statistics.[citation needed]

Effect of transformations[edit]


Bias, variance and mean squared error[edit]







\begin{align} \operatorname{MSE}(\hat{\theta})= & (\operatorname{E}[\hat{\theta}]-\theta)^2 + \operatorname{E}[\,(\hat{\theta} - \operatorname{E}[\,\hat{\theta}\,])^2\,]\\ = & (\operatorname{Bias}(\hat{\theta},\theta))^2 + \operatorname{Var}(\hat{\theta}) \end{align}

When the parameter is a vector, an analogous decomposition applies:[4]

\operatorname{MSE}(\hat{\theta }) =\operatorname{trace}(\operatorname{Var}(\hat{\theta })) +\left\Vert\operatorname{Bias}(\hat{\theta},\theta) \right\Vert^{2}



is the trace of the covariance matrix of the estimator.


Example: Estimation of population variance[edit]

For example,[5] suppose an estimator of the form

T^2 = c \sum_{i=1}^n\left(X_i-\overline{X}\,\right)^2 = c n S^2


\begin{align}\operatorname{MSE} = & \operatorname{E}\left[(T^2 - \sigma^2)^2\right] \\ = & \left(\operatorname{E}\left[T^2 - \sigma^2\right]\right)^2 + \operatorname{Var}(T^2)\end{align}


\operatorname{E}[nS^2] = (n-1)\sigma^2\text{ and }\operatorname{Var}(nS^2)=2(n-1)\sigma^4.

and so

\operatorname{MSE} = (c (n-1) - 1)^2\sigma^4 + 2c^2(n-1)\sigma^4




Bayesian view[edit]



p(\theta \mid D, I) \propto p(\theta \mid I) p(D \mid \theta, I)




\operatorname{Expected Loss} = \operatorname{E}\left[\left(c n S^2 - \sigma^2\right)^2\right] = \operatorname{E}\left[\sigma^4 \left(c n \tfrac{S^2}{\sigma^2} -1 \right)^2\right]

A standard choice of uninformative prior for this problem is the Jeffreys prior, \scriptstyle{p(\sigma^2) \;\propto\; 1 / \sigma^2}, which is equivalent to adopting a rescaling-invariant flat prior for ln( σ2).


p\left(\tfrac{S^2}{\sigma^2}\mid S^2\right) = p\left(\tfrac{S^2}{\sigma^2}\mid \sigma^2\right) = g\left(\tfrac{S^2}{\sigma^2}\right)

However, whilst

\operatorname{E}_{p(S^2\mid \sigma^2)}\left[\sigma^4 \left(c n \tfrac{S^2}{\sigma^2} -1 \right)^2\right] = \sigma^4 \operatorname{E}_{p(S^2\mid \sigma^2)}\left[\left(c n \tfrac{S^2}{\sigma^2} -1 \right)^2\right]

in contrast

\operatorname{E}_{p(\sigma^2\mid S^2)}\left[\sigma^4 \left(c n \tfrac{S^2}{\sigma^2} -1 \right)^2\right] \neq \sigma^4 \operatorname{E}_{p(\sigma^2\mid S^2)}\left[\left(c n \tfrac{S^2}{\sigma^2} -1 \right)^2\right]




See also[edit]


  2. ^ Hardy, M. (1 March 2003). "An Illuminating Counterexample". American Mathematical Monthly 110 (3): 234–238. doi:10.2307/3647938. ISSN 0002-9890. JSTOR 3647938.  edit
  3. ^ Brown (1947), page 583
  4. ^ Taboga, Marco (2010). "Mean squared error". Lectures on probability theory and mathematical statistics. 
    But cf e.g. discussion in Casella and Berger (2001), Statistical Inference (2nd edition), Duxbury. ISBN 0534243126. P. 332.


External links[edit]