G07 Chapter Contents
G07 Chapter Introduction (PDF version)
NAG Library Manual

NAG Library Chapter Introduction

G07 – Univariate Estimation

+ Contents

1  Scope of the Chapter


2  Background to the Problems

Statistical inference is concerned with the making of inferences about a population using the observed part of the population called a sample. The population can usually be described using a probability model which will be written in terms of some unknown parameters. For example, the hours of relief given by a drug may be assumed to follow a Normal distribution with mean μ and variance σ2; it is then required to make inferences about the parameters, μ and σ2, on the basis of an observed sample of relief times.
There are two main aspects of statistical inference: the estimation of the parameters and the testing of hypotheses about the parameters. In the example above, the values of the parameter σ2 may be estimated and the hypothesis that μ3 tested. This chapter is mainly concerned with estimation but the test of a hypothesis about a parameter is often closely linked to its estimation. Tests of hypotheses which are not linked closely to estimation are given in the chapter on nonparametric statistics (Chapter G08).

2.1  Maximum Likelihood Estimation

Let Xi be a univariate random variable with probability density function
where θ is a vector of length p consisting of the unknown parameters. For example, a Normal distribution with mean θ1 and standard deviation θ2 has probability density function
12πθ2 exp-12 xi-θ1θ2 2 .
The likelihood for a sample of n independent observations is
Like=i=1nfXi xi;θ ,
where xi is the observed value of Xi. If each Xi has an identical distribution, this reduces to
Like=i=1nfX xi;θ , (1)
and the log-likelihood is
logLike=L=i=1nlogfXxi;θ. (2)
The maximum likelihood estimates (θ^) of θ are the values of θ that maximize (1) and (2). If the range of X is independent of the parameters, then θ^ can usually be found as the solution to
i=1n θ^j logfXxi;θ^= L θ^j =0,  j=1,2,,p. (3)
Note that L θj  is known as the efficient score.
Maximum likelihood estimators possess several important properties.
E L θ =0, (4)
Iθ=-E 2L θ2 =E L θ 2 , (5)
and then that θ^ is asymptotically Normal with mean vector θ0 and variance-covariance matrix Iθ0-1 where θ0 denotes the true value of θ. The matrix Iθ is known as the information matrix and Iθ0-1 is known as the Cramer–Rao lower bound for the variance of an estimator of θ.
For example, if we consider a sample, x1,x2,,xn, of size n drawn from a Normal distribution with unknown mean μ and unknown variance σ2 then we have
L=logLikeμ,σ2;x=-n2log2π-n2logσ2-i=1n xi-μ 2/2σ2
and thus
L μ =i= 1n xi-μ/σ2
L σ2 =-n2σ2 +i=1n xi-μ 2/2σ4.
σ^2=i=1n xi-x- 2/n.
These maximum likelihood estimates are asymptotically Normal with mean vector a, where
and covariance matrix C. To obtain C we find the second derivatives of L with respect to μ and σ2 as follows:
2L μ2 =- nσ2 2L σ22 = n2σ4 -i=1n xi-μ 2/σ6 2L μσ2 = 2L σ2μ =- nx--μσ4.
C-1=-E 2 L μ2 2 L σ2μ 2 L μσ2 2 L σ22 = n/σ2 0 0 n/2σ4
so that
C= σ2/n 0 0 2σ4/n .
To obtain an estimate of C the matrix may be evaluated at the maximum likelihood estimates.
The use of maximum likelihood estimation allows the construction of generalized likelihood ratio tests. If λ=2l1-l2, where l1 is the maximized log-likelihood function for a model 1 and l2 is the maximized log-likelihood function for a model 2, then under the hypothesis that model 2 is correct, 2λ is asymptotically distributed as a χ2 variable with p-q degrees of freedom. Consider two models in which model 1 has p parameters and model 2 is a sub-model (nested model) of model 1 with q<p parameters, that is model 1 has an extra p-q parameters. This result provides a useful method for performing hypothesis tests on the parameters. Alternatively, tests exist based on the asymptotic Normality of the estimator and the efficient score; see page 315 of Cox and Hinkley (1974).

2.2  Confidence Intervals

Suppose we can find a function, tx,θ, whose distribution depends upon the sample x but not on the unknown parameter θ, and which is a monotonic (say decreasing) function in θ for each x, then we can find t1 such that P t 1 t x,θ = 1 - α  no matter what θ happens to be. The function tx,θ is known as a pivotal quantity. Since the function is monotonic the statement that t1tx,θ may be rewritten as θθ1x see Figure 1. The statistic θ1x will vary from sample to sample and if we assert that θθ1x for any sample values which arise, we will be right in a proportion 1-α of the cases, in the long run or on average. We call θ1x a 1-α upper confidence limit for θ.
Figure 1
Figure 1
We have considered only an upper confidence limit. The above idea may be generalized to a two-sided confidence interval where two quantities, t0 and t1, are found such that for all θ, P t 1 t x,θ t 0 = 1 - α . This interval may be rewritten as θ0xθθ1x. Thus if we assert that θ lies in the interval [θ0x,θ1x] we will be right on average in 1-α proportion of the times under repeated sampling.
Hypothesis (significance) tests on the parameters may be used to find these confidence limits. For example, if we observe a value, k, from a binomial distribution, with known parameter n and unknown parameter p, then to find the lower confidence limit we find pl such that the probability that the null hypothesis H0: p=pl (against the one sided alternative that p>pl) will be rejected, is less than or equal to α/2. Thus for a binomial random variable, B, with parameters n and pl we require that PBkα/2. The upper confidence limit, pu, can be constructed in a similar way.

2.3  Robust Estimation

For particular cases the probability density function can be written as
fXixi;θ=1θ2g xi-θ1θ2
for a suitable function g; then θ1 is known as a location parameter and θ2, usually written as σ, is known as a scale parameter. This is true of the Normal distribution.
If θ1 is a location parameter, as described above, then equation (3) becomes
i=1nψ xi-θ^1σ^=0, (6)
where ψz=- ddz loggz.
For the scale parameter σ (or σ2) the equation is
i=1nχ xi-θ^1σ^=n/2, (7)
where χz=zψz/2.
For the Normal distribution ψz=z and χz=z2/2. Thus, the maximum likelihood estimates for θ1 and σ2 are the sample mean and variance with the n divisor respectively. As the latter is biased, (7) can be replaced by
i=1nχ xi-θ^1σ^=n-1β, (8)
where β is a suitable constant, which for the Normal χ function is 12 .
The influence of an observation on the estimates depends on the form of the ψ and χ functions. For a discussion of influence, see Hampel et al. (1986) and Huber (1981). The influence of extreme values can be reduced by bounding the values of the ψ- and χ-functions. One suggestion due to Huber (1981) is
ψz= -C, ||z<-C z, zC C, ||z>C.
Figure 2
Figure 2
Redescending ψ-functions are often considered; these give zero values to ψz for large positive or negative values of z. Hampel et al. (1986) suggested
ψz= -ψ-z z, 0zh1. h1, h1zh2. h1h3-z/h3-h2, h2zh3. 0, z>h3.
Figure 3
Figure 3
Usually a χ-function based on Huber's ψ-function is used: χ=ψ2/2. Estimators based on such bounded ψ-functions are known as M-estimators, and provide one type of robust estimator.
Other robust estimators for the location parameter are
(i) the sample median,
For the scale parameter, alternative estimators are

2.4  Robust Confidence Intervals


3  Recommendations on Choice and Use of Available Routines

Maximum Likelihood Estimation and Confidence Intervals
G07AAFprovides a confidence interval for the parameter p of the binomial distribution.
G07ABFprovides a confidence interval for the mean parameter of the Poisson distribution.
G07BFFprovides maximum likelihood estimates and their standard errors for the parameters of the generalized Pareto distribution.
Robust Estimation
G07DBFprovides M-estimates for location and, optionally, scale using four common forms of the ψ-function.
G07DCFproduces the M-estimates for location and, optionally, scale but for user-supplied ψ- and χ-functions.
Robust Internal Estimation
G07EAFproduces a rank based confidence interval for locations.
G07EBFproduces a rank based confidence interval for the difference in location between two populations.
Outlier Detection
This chapter provides two routines for identifying potential outlying values, G07GAF and G07GBF. Many of the model fitting routines, for examples those in Chapters G02 and G13 also return vectors of residuals which can also be used to aid in the identification of outlying values.

4  Functionality Index

2 sample t-test G07CAF
Confidence intervals for parameters, 
    binomial distribution G07AAF
    Poisson distribution G07ABF
Maximum likelihood estimation of parameters, 
    Normal distribution, grouped and/or censored data G07BBF
    Weibull distribution G07BEF
Outlier detection, 
        raw data or single variance supplied G07GAF
        two variances supplied G07GBF
Parameter estimates, 
    generalized Pareto distribution G07BFF
Robust estimation, 
    confidence intervals, 
        one sample G07EAF
        two samples G07EBF
    median, median absolute deviation and robust standard deviation G07DAF
    M-estimates for location and scale parameters, 
        standard weight functions G07DBF
        trimmed and winsorized means and estimates of their variance G07DDF
        user-defined weight functions G07DCF

5  Auxiliary Routines Associated with Library Routine Parameters


6  Routines Withdrawn or Scheduled for Withdrawal


7  References

Huber P J (1981) Robust Statistics Wiley
Silvey S D (1975) Statistical Inference Chapman and Hall

G07 Chapter Contents
G07 Chapter Introduction (PDF version)
NAG Library Manual

© The Numerical Algorithms Group Ltd, Oxford, UK. 2012