DOMAIN: MilDly COnservative Model-BAsed OfflINe Reinforcement Learning
Xiao-Yin Liu Xiao-Hu Zhou Xiao-Liang Xie Shi-Qi Liu Zhen-Qiu Feng Hao Li Mei-Jiang Gui Tian-Yu Xiang De-Xing Huang Zeng-Guang Hou
Model-based reinforcement learning (RL), which learns environment model from
offline dataset and generates more out-of-distribution model data, has become
an effective approach to the problem of distribution shift in offline RL. Due
to the gap between the learned and actual environment, conservatism should be
incorporated into the algorithm to balance accurate offline data and imprecise
model data. The conservatism of current algorithms mostly relies on model
uncertainty estimation. However, uncertainty estimation is unreliable and leads
to poor performance in certain scenarios, and the previous methods ignore
differences between the model data, which brings great conservatism. Therefore,
this paper proposes a milDly cOnservative Model-bAsed offlINe RL algorithm
(DOMAIN) without estimating model uncertainty to address the above issues.
DOMAIN introduces adaptive sampling distribution of model samples, which can
adaptively adjust the model data penalty. In this paper, we theoretically
demonstrate that the Q value learned by the DOMAIN outside the region is a
lower bound of the true Q value, the DOMAIN is less conservative than previous
model-based offline RL algorithms and has the guarantee of security policy
improvement. The results of extensive experiments show that DOMAIN outperforms
prior RL algorithms on the D4RL dataset benchmark, and achieves better
performance than other RL algorithms on tasks that require generalization.