Offline Policy Evaluation for Reinforcement Learning with Adaptively
  Collected Data
Sunil Madhow Dan Xiao Ming Yin Yu-Xiang Wang
Developing theoretical guarantees on the sample complexity of offline RL
methods is an important step towards making data-hungry RL algorithms
practically viable. Currently, most results hinge on unrealistic assumptions
about the data distribution -- namely that it comprises a set of i.i.d.
trajectories collected by a single logging policy. We consider a more general
setting where the dataset may have been gathered adaptively. We develop theory
for the TMIS Offline Policy Evaluation (OPE) estimator in this generalized
setting for tabular MDPs, deriving high-probability, instance-dependent bounds
on its estimation error. We also recover minimax-optimal offline learning in
the adaptive setting. Finally, we conduct simulations to empirically analyze
the behavior of these estimators under adaptive and non-adaptive regimes.