Online Estimation and Inference for Robust Policy Evaluation in
  Reinforcement Learning
Weidong Liu Jiyuan Tu Yichen Zhang Xi Chen
Recently, reinforcement learning has gained prominence in modern statistics,
with policy evaluation being a key component. Unlike traditional machine
learning literature on this topic, our work places emphasis on statistical
inference for the parameter estimates computed using reinforcement learning
algorithms. While most existing analyses assume random rewards to follow
standard distributions, limiting their applicability, we embrace the concept of
robust statistics in reinforcement learning by simultaneously addressing issues
of outlier contamination and heavy-tailed rewards within a unified framework.
In this paper, we develop an online robust policy evaluation procedure, and
establish the limiting distribution of our estimator, based on its Bahadur
representation. Furthermore, we develop a fully-online procedure to efficiently
conduct statistical inference based on the asymptotic distribution. This paper
bridges the gap between robust statistics and statistical inference in
reinforcement learning, offering a more versatile and reliable approach to
policy evaluation. Finally, we validate the efficacy of our algorithm through
numerical experiments conducted in real-world reinforcement learning
experiments.