Marginal Density Ratio for Off-Policy Evaluation in Contextual Bandits
Muhammad Faaiz Taufiq Arnaud Doucet Rob Cornish Jean-Francois Ton
Off-Policy Evaluation (OPE) in contextual bandits is crucial for assessing
new policies using existing data without costly experimentation. However,
current OPE methods, such as Inverse Probability Weighting (IPW) and Doubly
Robust (DR) estimators, suffer from high variance, particularly in cases of low
overlap between target and behavior policies or large action and context
spaces. In this paper, we introduce a new OPE estimator for contextual bandits,
the Marginal Ratio (MR) estimator, which focuses on the shift in the marginal
distribution of outcomes $Y$ instead of the policies themselves. Through
rigorous theoretical analysis, we demonstrate the benefits of the MR estimator
compared to conventional methods like IPW and DR in terms of variance
reduction. Additionally, we establish a connection between the MR estimator and
the state-of-the-art Marginalized Inverse Propensity Score (MIPS) estimator,
proving that MR achieves lower variance among a generalized family of MIPS
estimators. We further illustrate the utility of the MR estimator in causal
inference settings, where it exhibits enhanced performance in estimating
Average Treatment Effects (ATE). Our experiments on synthetic and real-world
datasets corroborate our theoretical findings and highlight the practical
advantages of the MR estimator in OPE for contextual bandits.