Efficient Two-Phase Offline Deep Reinforcement Learning from Preference
  Feedback
Yinglun Xu Gagandeep Singh
In this work, we consider the offline preference-based reinforcement learning
problem. We focus on the two-phase learning approach that is prevalent in
previous reinforcement learning from human preference works. We find a
challenge in applying two-phase learning in the offline PBRL setting that the
learned utility model can be too hard for the learning agent to optimize during
the second learning phase. To overcome the challenge, we propose a two-phasing
learning approach under behavior regularization through action clipping. The
insight is that the state-actions which are poorly covered by the dataset can
only provide limited information and increase the complexity of the problem in
the second learning phase. Our method ignores such state-actions during the
second learning phase to achieve higher learning efficiency. We empirically
verify that our method has high learning efficiency on a variety of datasets in
robotic control environments.