Bayesian Off-Policy Evaluation and Learning for Large Action Spaces
Imad Aouali Victor-Emmanuel Brunel David Rohde Anna Korba
In interactive systems, actions are often correlated, presenting an
opportunity for more sample-efficient off-policy evaluation (OPE) and learning
(OPL) in large action spaces. We introduce a unified Bayesian framework to
capture these correlations through structured and informative priors. In this
framework, we propose sDM, a generic Bayesian approach designed for OPE and
OPL, grounded in both algorithmic and theoretical foundations. Notably, sDM
leverages action correlations without compromising computational efficiency.
Moreover, inspired by online Bayesian bandits, we introduce Bayesian metrics
that assess the average performance of algorithms across multiple problem
instances, deviating from the conventional worst-case assessments. We analyze
sDM in OPE and OPL, highlighting the benefits of leveraging action
correlations. Empirical evidence showcases the strong performance of sDM.