Adaptive Instrument Design for Indirect Experiments
Yash Chandak Shiv Shankar Vasilis Syrgkanis Emma Brunskill
Indirect experiments provide a valuable framework for estimating treatment
effects in situations where conducting randomized control trials (RCTs) is
impractical or unethical. Unlike RCTs, indirect experiments estimate treatment
effects by leveraging (conditional) instrumental variables, enabling estimation
through encouragement and recommendation rather than strict treatment
assignment. However, the sample efficiency of such estimators depends not only
on the inherent variability in outcomes but also on the varying compliance
levels of users with the instrumental variables and the choice of estimator
being used, especially when dealing with numerous instrumental variables. While
adaptive experiment design has a rich literature for direct experiments, in
this paper we take the initial steps towards enhancing sample efficiency for
indirect experiments by adaptively designing a data collection policy over
instrumental variables. Our main contribution is a practical computational
procedure that utilizes influence functions to search for an optimal data
collection policy, minimizing the mean-squared error of the desired
(non-linear) estimator. Through experiments conducted in various domains
inspired by real-world applications, we showcase how our method can
significantly improve the sample efficiency of indirect experiments.