Statistical power

From Infogalactic: the planetary knowledge core
Jump to: navigation, search


 \mbox{power} = \mathbb P\big( \mbox{reject } H_0 \big| H_1 \mbox{ is true} \big)

The power of a test sometimes, less formally, refers to the probability of rejecting the null when it is not correct, though this is not the formal definition stated above. The power is in general a function of the possible distributions, often determined by a parameter, under the alternative hypothesis. As the power increases, there are decreasing chances of a Type II error (false negative), which are also referred to as the false negative rate (β) since the power is equal to 1−β, again, under the alternative hypothesis. A similar concept is Type I error, also referred to as the “false positive rate” or the level of a test under the null hypothesis.

Power analysis can be used to calculate the minimum sample size required so that one can be reasonably likely to detect an effect of a given size. For example: “how many times do I need to toss a coin to conclude it is rigged?”[1] Power analysis can also be used to calculate the minimum effect size that is likely to be detected in a study using a given sample size. In addition, the concept of power is used to make comparisons between different statistical testing procedures: for example, between a parametric and a nonparametric test of the same hypothesis.



Factors influencing power

Statistical power may depend on a number of factors. Some factors may be particular to a specific testing situation, but at a minimum, power nearly always depends on the following three factors:

A significance criterion is a statement of how unlikely a positive result must be, if the null hypothesis of no effect is true, for the null hypothesis to be rejected. The most commonly used criteria are probabilities of 0.05 (5%, 1 in 20), 0.01 (1%, 1 in 100), and 0.001 (0.1%, 1 in 1000). If the criterion is 0.05, the probability of the data implying an effect at least as large as the observed effect when the null hypothesis is true must be less than 0.05, for the null hypothesis of no effect to be rejected. One easy way to increase the power of a test is to carry out a less conservative test by using a larger significance criterion, for example 0.10 instead of 0.05. This increases the chance of rejecting the null hypothesis (i.e. obtaining a statistically significant result) when the null hypothesis is false, that is, reduces the risk of a Type II error (false negative regarding whether an effect exists). But it also increases the risk of obtaining a statistically significant result (i.e. rejecting the null hypothesis) when the null hypothesis is not false; that is, it increases the risk of a Type I error (false positive).


The sample size determines the amount of sampling error inherent in a test result. Other things being equal, effects are harder to detect in smaller samples. Increasing sample size is often the easiest way to boost the statistical power of a test. How increased sample size translates to higher power is a measure of the efficiency of the test—for example, the sample size required for a given power.[2]

The precision with which the data are measured also influences statistical power. Consequently, power can often be improved by reducing the measurement error in the data. A related concept is to improve the “reliability” of the measure being assessed (as in psychometric reliability).

The design of an experiment or observational study often influences the power. For example, in a two-sample testing situation with a given total sample size n, it is optimal to have equal numbers of observations from the two populations being compared (as long as the variances in the two populations are the same). In regression analysis and Analysis of Variance, there are extensive theories and practical strategies for improving the power based on optimally setting the values of the independent variables in the model.


Although there are no formal standards for power (sometimes referred to as π), most researchers assess the power of their tests using π=0.80 as a standard for adequacy. This convention implies a four-to-one trade off between β-risk and α-risk. (β is the probability of a Type II error; α is the probability of a Type I error, 0.2 and 0.05 are conventional values for β and α). However, there will be times when this 4-to-1 weighting is inappropriate. In medicine, for example, tests are often designed in such a way that no false negatives (Type II errors) will be produced. But this inevitably raises the risk of obtaining a false positive (a Type I error). The rationale is that it is better to tell a healthy patient “we may have found something—let's test further,” than to tell a diseased patient “all is well.”[3]

Power analysis is appropriate when the concern is with the correct rejection, or not, of a null hypothesis. In many contexts, the issue is less about determining if there is or is not a difference but rather with getting a more refined estimate of the population effect size. For example, if we were expecting a population correlation between intelligence and job performance of around 0.50, a sample size of 20 will give us approximately 80% power (alpha = 0.05, two-tail) to reject the null hypothesis of zero correlation. However, in doing this study we are probably more interested in knowing whether the correlation is 0.30 or 0.60 or 0.50. In this context we would need a much larger sample size in order to reduce the confidence interval of our estimate to a range that is acceptable for our purposes. Techniques similar to those employed in a traditional power analysis can be used to determine the sample size required for the width of a confidence interval to be less than a given value.

Many statistical analyses involve the estimation of several unknown quantities. In simple cases, all but one of these quantities is a nuisance parameter. In this setting, the only relevant power pertains to the single quantity that will undergo formal statistical inference. In some settings, particularly if the goals are more "exploratory", there may be a number of quantities of interest in the analysis. For example, in a multiple regression analysis we may include several covariates of potential interest. In situations such as this where several hypotheses are under consideration, it is common that the powers associated with the different hypotheses differ. For instance, in multiple regression analysis, the power for detecting an effect of a given size is related to the variance of the covariate. Since different covariates will have different variances, their powers will differ as well.


It is also important to consider the statistical power of a hypothesis test when interpreting its results. A test's power is the probability of correctly rejecting the null hypothesis when it is false; a test's power is influenced by the choice of significance level for the test, the size of the effect being measured, and the amount of data available. A hypothesis test may fail to reject the null, for example, if a true difference exists between two populations being compared by a t-test but the effect is small and the sample size is too small to distinguish the effect from random chance.[4] Many clinical trials, for instance, have low statistical power to detect differences in adverse effects of treatments, since such effects may be rare and the number of affected patients small.[5]

A priori vs. post hoc analysis

Power analysis can either be done before (a priori or prospective power analysis) or after (post hoc or retrospective power analysis) data are collected. A priori power analysis is conducted prior to the research study, and is typically used in estimating sufficient sample sizes to achieve adequate power. Post-hoc power analysis is conducted after a study has been completed, and uses the obtained sample size and effect size to determine what the power was in the study, assuming the effect size in the sample is equal to the effect size in the population. Whereas the utility of prospective power analysis in experimental design is universally accepted, the usefulness of retrospective techniques is controversial.[6] Falling for the temptation to use the statistical analysis of the collected data to estimate the power will result in uninformative and misleading values. In particular, it has been shown [7] that post-hoc power in its simplest form is a one-to-one function of the p-value attained. This has been extended[7] to show that all post-hoc power analyses suffer from what is called the "power approach paradox" (PAP), in which a study with a null result is thought to show MORE evidence that the null hypothesis is actually true when the p-value is smaller, since the apparent power to detect an actual effect would be higher. In fact, a smaller p-value is properly understood to make the null hypothesis LESS likely to be true.


Funding agencies, ethics boards and research review panels frequently request that a researcher perform a power analysis, for example to determine the minimum number of animal test subjects needed for an experiment to be informative. In frequentist statistics, an underpowered study is unlikely to allow one to choose between hypotheses at the desired significance level. In Bayesian statistics, hypothesis testing of the type used in classical power analysis is not done. In the Bayesian framework, one updates his or her prior beliefs using the data obtained in a given study. In principle, a study that would be deemed underpowered from the perspective of hypothesis testing could still be used in such an updating process. However, power remains a useful measure of how much a given experiment size can be expected to refine one's beliefs. A study with low power is unlikely to lead to a large change in beliefs.


The following is an example that shows how to compute power for a randomized experiment: Suppose the goal of an experiment is to study the effect of a treatment on some quantity, and compare research subjects by measuring the quantity before and after the treatment, analyzing the data using a paired t-test. Let A_i and B_i denote the pre-treatment and post-treatment measures on subject i respectively. The possible effect of the treatment should be visible in the differences D_i=B_i-A_i, which are assumed to be independently distributed, all with the same expected value and variance.

'The effect of the treatment can be analyzed using a one-sided t-test. The null hypothesis of no effect will be that the mean difference will be zero, i.e. H_0: \mu_D=0. In this case, the alternative hypothesis states a positive effect, corresponding to H_1: \mu_D>0. The test statistic is:


where n is the sample size, \bar{D}_n=\frac{1}{n}\sum_{i=1}^n D_i, and \hat{\sigma}_D/\sqrt{n} is the standard error. The distribution of the test statistic under the null hypothesis follows a Student t-distribution. Furthermore, assume that the null hypothesis will be rejected at the significance level of  \alpha=0.05 . Since n is large, one can approximate the t-distribution by a normal distribution and calculate the critical value using the quantile function \Phi of the normal distribution. It turns out that the null hypothesis will be rejected if

T_n > 1.64.

Now suppose that the alternative hypothesis is true and \mu_D=\theta. Then, the power is

B(\theta)&=&P(T_n > 1.64|\mu_D=\theta) \\
&=&P(\frac{\bar{D}_n-0}{\hat{\sigma}_D/\sqrt{n} }> 1.64|\mu_D=\theta) \\
 &=&P\left(\frac{\bar{D}_n-\theta+\theta}{\hat{\sigma}_D/\sqrt{n}} > 1.64\right|\mu_D=\theta)\\
&=& P\left(\frac{\bar{D}_n-\theta}{\hat{\sigma}_D/\sqrt{n}} > 1.64-\frac{\theta}{\hat{\sigma}_D/\sqrt{n}}\right|\mu_D=\theta)\\
&=& 1- P\left(\frac{\bar{D}_n-\theta}{\hat{\sigma}_D/\sqrt{n}} < 1.64-\frac{\theta}{\hat{\sigma}_D/\sqrt{n}}\right|\mu_D=\theta)\\

For large n, T_n approximately follows a standard normal distribution when the alternative hypothesis is true, the approximate power can be calculated as

B(\theta)\approx 1-\Phi(1.64-\frac{\theta}{\hat{\sigma}_D/\sqrt{n}}).

According to this formula, the power increases with the values of the parameter \theta. For a specific value of \theta a higher power may be obtained by increasing the sample size n.

It is not possible to guarantee a sufficient large power for all values of \theta, as \theta may be very close to 0. The minimum (infimum) value of the power is equal to the size of the test, \alpha, in this example 0.05. However, it is of no importance to distinguish between \theta=0 and small positive values. If it is desirable to have enough power, say at least 0.90, to detect values of \theta >1, the required sample size can be calculated approximately:

B(1)\approx 1-\Phi(1.64-\sqrt{n}/\hat{\sigma}_D) >0{.}90\ ,

from which it follows that

\Phi(1.64-\sqrt{n}/\hat{\sigma}_D) <0.10.


\sqrt{n}/\hat{\sigma}_D > 1.64- z_{0.10}=1.64+1.28\approx 3



where z_{0.10} is a standard normal quantile; see Probit for an explanation of the relationship between \Phi and z-values.

Software for power and sample size calculations

Numerous programs are available for performing power and sample size calculations. These include commercial software

and free software


Bayesian power

In Frequentist setting, parameter is assumed to have a specific value which is unlikely to be true. This issue can be addressed by assuming the parameter has a distribution. The resulted power is sometimes referred to as Bayesian power which is commonly used in clinical trial design.

Predictive probability of success

Both Frequentist power and Bayesian power uses statistical significance as success criteria. However statistical significance is often not enough to define success. To address this issue, the power concept can be extended to the concept of predictive probability of success (PPOS). The success criteria for PPOS is not restricted to statistical significance and is commonly used in clinical trial designs.

See also


  2. Everitt 2002, p. 321.
  4. Ellis, Paul (2010). The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results. Cambridge University Press. p. 52. ISBN 978-0521142465. 
  5. Tsang, R.; Colley, L.; Lynd, L. D. (2009). "Inadequate statistical power to detect clinically significant differences in adverse event rates in randomized controlled trials". Journal of Clinical Epidemiology. 62 (6): 609–616. PMID 19013761. doi:10.1016/j.jclinepi.2008.08.005. 
  7. 7.0 7.1 Hoenig and Heisey (2001)The Abuse of PowerThe American Statistician 55(1):19-24 [1]


  • Everitt, Brian S. (2002). The Cambridge Dictionary of Statistics. Cambridge University Press. ISBN 0-521-81099-X. 
  • Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2nd ed.). ISBN 0-8058-0283-5. 
  • Aberson, C. L. (2010). Applied Power Analysis for the Behavioral Science. ISBN 1-84872-835-2. 

External links

Further Explanations