Offline Reinforcement Learning with On-Policy Q-Function Regularization
Laixi Shi Robert Dadashi Yuejie Chi Pablo Samuel Castro Matthieu Geist
The core challenge of offline reinforcement learning (RL) is dealing with the
(potentially catastrophic) extrapolation error induced by the distribution
shift between the history dataset and the desired policy. A large portion of
prior work tackles this challenge by implicitly/explicitly regularizing the
learning policy towards the behavior policy, which is hard to estimate reliably
in practice. In this work, we propose to regularize towards the Q-function of
the behavior policy instead of the behavior policy itself, under the premise
that the Q-function can be estimated more reliably and easily by a SARSA-style
estimate and handles the extrapolation error more straightforwardly. We propose
two algorithms taking advantage of the estimated Q-function through
regularizations, and demonstrate they exhibit strong performance on the D4RL
benchmarks.