An Analysis of Switchback Designs in Reinforcement Learning
Qianglin Wen Chengchun Shi Ying Yang Niansheng Tang Hongtu Zhu
This paper offers a detailed investigation of switchback designs in A/B
testing, which alternate between baseline and new policies over time. Our aim
is to thoroughly evaluate the effects of these designs on the accuracy of their
resulting average treatment effect (ATE) estimators. We propose a novel "weak
signal analysis" framework, which substantially simplifies the calculations of
the mean squared errors (MSEs) of these ATEs in Markov decision process
environments. Our findings suggest that (i) when the majority of reward errors
are positively correlated, the switchback design is more efficient than the
alternating-day design which switches policies in a daily basis. Additionally,
increasing the frequency of policy switches tends to reduce the MSE of the ATE
estimator. (ii) When the errors are uncorrelated, however, all these designs
become asymptotically equivalent. (iii) In cases where the majority of errors
are negative correlated, the alternating-day design becomes the optimal choice.
These insights are crucial, offering guidelines for practitioners on designing
experiments in A/B testing. Our analysis accommodates a variety of policy value
estimators, including model-based estimators, least squares temporal difference
learning estimators, and double reinforcement learning estimators, thereby
offering a comprehensive understanding of optimal design strategies for policy
evaluation in reinforcement learning.