State-Action Similarity-Based Representations for Off-Policy Evaluation
Brahma S. Pavse Josiah P. Hanna
In reinforcement learning, off-policy evaluation (OPE) is the problem of
estimating the expected return of an evaluation policy given a fixed dataset
that was collected by running one or more different policies. One of the more
empirically successful algorithms for OPE has been the fitted q-evaluation
(FQE) algorithm that uses temporal difference updates to learn an action-value
function, which is then used to estimate the expected return of the evaluation
policy. Typically, the original fixed dataset is fed directly into FQE to learn
the action-value function of the evaluation policy. Instead, in this paper, we
seek to enhance the data-efficiency of FQE by first transforming the fixed
dataset using a learned encoder, and then feeding the transformed dataset into
FQE. To learn such an encoder, we introduce an OPE-tailored state-action
behavioral similarity metric, and use this metric and the fixed dataset to
learn an encoder that models this metric. Theoretically, we show that this
metric allows us to bound the error in the resulting OPE estimate. Empirically,
we show that other state-action similarity metrics lead to representations that
cannot represent the action-value function of the evaluation policy, and that
our state-action representation method boosts the data-efficiency of FQE and
lowers OPE error relative to other OPE-based representation learning methods on
challenging OPE tasks. We also empirically show that the learned
representations significantly mitigate divergence of FQE under varying
distribution shifts. Our code is available here:
https://github.com/Badger-RL/ROPE.