Learning Metrics that Maximise Power for Accelerated A/B-Tests
Olivier Jeunen Aleksei Ustimenko
Online controlled experiments are a crucial tool to allow for confident
decision-making in technology companies. A North Star metric is defined (such
as long-term revenue or user retention), and system variants that statistically
significantly improve on this metric in an A/B-test can be considered superior.
North Star metrics are typically delayed and insensitive. As a result, the cost
of experimentation is high: experiments need to run for a long time, and even
then, type-II errors (i.e. false negatives) are prevalent.
  We propose to tackle this by learning metrics from short-term signals that
directly maximise the statistical power they harness with respect to the North
Star. We show that existing approaches are prone to overfitting, in that higher
average metric sensitivity does not imply improved type-II errors, and propose
to instead minimise the $p$-values a metric would have produced on a log of
past experiments. We collect such datasets from two social media applications
with over 160 million Monthly Active Users each, totalling over 153 A/B-pairs.
Empirical results show that we are able to increase statistical power by up to
78% when using our learnt metrics stand-alone, and by up to 210% when used in
tandem with the North Star. Alternatively, we can obtain constant statistical
power at a sample size that is down to 12% of what the North Star requires,
significantly reducing the cost of experimentation.