When is Offline Policy Selection Sample Efficient for Reinforcement
  Learning?
Vincent Liu Prabhat Nagarajan Andrew Patterson Martha White
Offline reinforcement learning algorithms often require careful
hyperparameter tuning. Consequently, before deployment, we need to select
amongst a set of candidate policies. As yet, however, there is little
understanding about the fundamental limits of this offline policy selection
(OPS) problem. In this work we aim to provide clarity on when sample efficient
OPS is possible, primarily by connecting OPS to off-policy policy evaluation
(OPE) and Bellman error (BE) estimation. We first show a hardness result, that
in the worst case, OPS is just as hard as OPE, by proving a reduction of OPE to
OPS. As a result, no OPS method can be more sample efficient than OPE in the
worst case. We then propose a BE method for OPS, called Identifiable BE
Selection (IBES), that has a straightforward method for selecting its own
hyperparameters. We highlight that using IBES for OPS generally has more
requirements than OPE methods, but if satisfied, can be more sample efficient.
We conclude with an empirical study comparing OPE and IBES, and by showing the
difficulty of OPS on an offline Atari benchmark dataset.