Mildly Constrained Evaluation Policy for Offline Reinforcement Learning
Linjie Xu Zhengyao Jiang Jinyu Wang Lei Song Jiang Bian
Offline reinforcement learning (RL) methodologies enforce constraints on the
policy to adhere closely to the behavior policy, thereby stabilizing value
learning and mitigating the selection of out-of-distribution (OOD) actions
during test time. Conventional approaches apply identical constraints for both
value learning and test time inference. However, our findings indicate that the
constraints suitable for value estimation may in fact be excessively
restrictive for action selection during test time. To address this issue, we
propose a Mildly Constrained Evaluation Policy (MCEP) for test time inference
with a more constrained target policy for value estimation. Since the target
policy has been adopted in various prior approaches, MCEP can be seamlessly
integrated with them as a plug-in. We instantiate MCEP based on TD3-BC
[Fujimoto and Gu, 2021] and AWAC [Nair et al., 2020] algorithms. The empirical
results on MuJoCo locomotion tasks show that the MCEP significantly outperforms
the target policy and achieves competitive results to state-of-the-art offline
RL methods. The codes are open-sourced at https://github.com/egg-west/MCEP.git.