From Wikipedia, the free encyclopedia
  (Redirected from Estimators)
Jump to: navigation, search





An "estimator" or "point estimate" is a statistic (that is, a function of the data) that is used to infer the value of an unknown parameter in a statistical model. The parameter being estimated is sometimes called the estimand. It can be either finite-dimensional (in parametric and semi-parametric models), or infinite-dimensional (semi-parametric and non-parametric models).[1] If the parameter is denoted θ then the estimator is traditionally written by adding a circumflex over the symbol: \scriptstyle\hat\theta. Being a function of the data, the estimator is itself a random variable; a particular realization of this random variable is called the "estimate". Sometimes the words "estimator" and "estimate" are used interchangeably.


When the word "estimator" is used without a qualifier, it usually refers to point estimation. The estimate in this case is a single point in the parameter space. Other types of estimators also exist: interval estimators, where the estimates are subsets of the parameter space.



Suppose there is a fixed parameter  \theta \ that needs to be estimated. Then an "estimator" is a function that maps the sample space to a set of sample estimates. An estimator of  \theta \ is usually denoted by the symbol \widehat{\theta}. It is often convenient to express the theory using the algebra of random variables: thus if X is used to denote a random variable corresponding to the observed data, the estimator (itself treated as a random variable) is symbolised as a function of that random variable, \widehat{\theta}(X). The estimate for a particular observed dataset (i.e. for X=x) is then \widehat{\theta}(x), which is a fixed value. Often an abbreviated notation is used in which \widehat{\theta} is interpreted directly as a random variable, but this can cause confusion.

Quantified properties[edit]

The following definitions and attributes are relevant.[2]


For a given sample  x \ , the "error" of the estimator \widehat{\theta} is defined as

e(x)=\widehat{\theta}(x) - \theta,


Mean squared error[edit]

The mean squared error of \widehat{\theta} is defined as the expected value (probability-weighted average, over all samples) of the squared errors; that is,

\operatorname{MSE}(\widehat{\theta}) = \operatorname{E}[(\widehat{\theta}(X) - \theta)^2].


Sampling deviation[edit]

For a given sample  x \ , the sampling deviation of the estimator \widehat{\theta} is defined as

d(x) =\widehat{\theta}(x) - \operatorname{E}( \widehat{\theta}(X) ) =\widehat{\theta}(x) - \operatorname{E}( \widehat{\theta} ),

where  \operatorname{E}( \widehat{\theta}(X) ) is the expected value of the estimator. Note that the sampling deviation, d, depends not only on the estimator, but on the sample.


The variance of \widehat{\theta} is simply the expected value of the squared sampling deviations; that is, \operatorname{var}(\widehat{\theta}) = \operatorname{E}[(\widehat{\theta} - \operatorname{E}(\widehat{\theta}) )^2]. It is used to indicate how far, on average, the collection of estimates are from the expected value of the estimates. Note the difference between MSE and variance. If the parameter is the bull's-eye of a target, and the arrows are estimates, then a relatively high variance means the arrows are dispersed, and a relatively low variance means the arrows are clustered. Some things to note: even if the variance is low, the cluster of arrows may still be far off-target, and even if the variance is high, the diffuse collection of arrows may still be unbiased. Finally, note that even if all arrows grossly miss the target, if they nevertheless all hit the same point, the variance is zero.


The bias of \widehat{\theta} is defined as B(\widehat{\theta}) = \operatorname{E}(\widehat{\theta}) - \theta. It is the distance between the average of the collection of estimates, and the single parameter being estimated. It also is the expected value of the error, since  \operatorname{E}(\widehat{\theta}) - \theta = \operatorname{E}(\widehat{\theta} - \theta ) . If the parameter is the bull's-eye of a target, and the arrows are estimates, then a relatively high absolute value for the bias means the average position of the arrows is off-target, and a relatively low absolute bias means the average position of the arrows is on target. They may be dispersed, or may be clustered. The relationship between bias and variance is analogous to the relationship between accuracy and precision.



Relationships among the quantities[edit]

  • The MSE, variance, and bias, are related: \operatorname{MSE}(\widehat{\theta}) = \operatorname{var}(\widehat\theta) + (B(\widehat{\theta}))^2, i.e. mean squared error = variance + square of bias. In particular, for an unbiased estimator, the variance equals the MSE.

Behavioural properties[edit]


Main article: Consistent estimator






Asymptotic normality[edit]

Main article: Asymptotic normality

An asymptotically normal estimator is a consistent estimator whose distribution around the true parameter θ approaches a normal distribution with standard deviation shrinking in proportion to 1/\sqrt{n} as the sample size n grows. Using \xrightarrow{D} to denote convergence in distribution, tn is asymptotically normal if

\sqrt{n}(t_n - \theta) \xrightarrow{D} N(0,V),

for some V.

In this formulation V/n can be called the asymptotic variance of the estimator. Note that convergence will not necessarily have occurred for any finite "n", therefore this value is only an approximation to the true variance of the estimator, while in the limit the asymptotic variance is simply zero. Stated a little more accurately, the distribution of the estimator tn converges weakly to a dirac delta function centered at \theta.

The central limit theorem implies asymptotic normality of the sample mean \bar X as an estimator of the true mean. More generally, maximum likelihood estimators are asymptotically normal under fairly weak regularity conditions — see the asymptotics section of the maximum likelihood article. However, not all estimators are asymptotically normal; the simplest examples are found when the true value of a parameter lies on the boundary of the allowable parameter region.






See: Robust estimator, Robust statistics

See also[edit]


  1. ^ Kosorok (2008), Section 3.1, pp 35–39.
  2. ^ Jaynes (2007), p.172.


External links[edit]