CROP: Conservative Reward for Model-based Offline Policy Optimization
Hao Li Xiao-Hu Zhou Xiao-Liang Xie Shi-Qi Liu Zhen-Qiu Feng Xiao-Yin Liu Mei-Jiang Gui Tian-Yu Xiang De-Xing Huang Bo-Xian Yao Zeng-Guang Hou
Offline reinforcement learning (RL) aims to optimize policy using collected
data without online interactions. Model-based approaches are particularly
appealing for addressing offline RL challenges due to their capability to
mitigate the limitations of offline data through data generation using models.
Prior research has demonstrated that introducing conservatism into the model or
Q-function during policy optimization can effectively alleviate the prevalent
distribution drift problem in offline RL. However, the investigation into the
impacts of conservatism in reward estimation is still lacking. This paper
proposes a novel model-based offline RL algorithm, Conservative Reward for
model-based Offline Policy optimization (CROP), which conservatively estimates
the reward in model training. To achieve a conservative reward estimation, CROP
simultaneously minimizes the estimation error and the reward of random actions.
Theoretical analysis shows that this conservative reward mechanism leads to a
conservative policy evaluation and helps mitigate distribution drift.
Experiments on D4RL benchmarks showcase that the performance of CROP is
comparable to the state-of-the-art baselines. Notably, CROP establishes an
innovative connection between offline and online RL, highlighting that offline
RL problems can be tackled by adopting online RL techniques to the empirical
Markov decision process trained with a conservative reward. The source code is
available with https://github.com/G0K0URURI/CROP.git.