What does "Peeking" mean?

Definition of Peeking in the context of A/B testing (online controlled experiments).

What is Peeking?

Aliases: optional stopping

"Peeking", or sometimes "unaccounted peeking" and more precisely "unaccounted peeking with intent to stop" is a term used to describe the poor practice of checking the results of an ongoing A/B test with the intent to stop it and make a decision or inference based on the observed outcome: value of the variable of interest, reaching a given significance threshold, etc. Peeking is a significant treat to the validity of any online controlled experiment as it can uncontrollably increase the type I error rate and render any significance calculations or confidence intervals meaningless.

It is a poor practice since it causes a discrepancy between the nominal p-value you calculate and the actual p-value which reflects the probability of observing the result you observed under the actual circumstances. With just a few looks at the data the actual significance can be orders of magnitude larger than the nominal.

Articles on Peeking


Purchase Statistical Methods in Online A/B Testing

Glossary Index by Letter

