STRAPPER: Preference-based Reinforcement Learning via Self-training
  Augmentation and Peer Regularization
Yachen Kang Li He Jinxin Liu Zifeng Zhuang Donglin Wang
Preference-based reinforcement learning (PbRL) promises to learn a complex
reward function with binary human preference. However, such human-in-the-loop
formulation requires considerable human effort to assign preference labels to
segment pairs, hindering its large-scale applications. Recent approache has
tried to reuse unlabeled segments, which implicitly elucidates the distribution
of segments and thereby alleviates the human effort. And consistency
regularization is further considered to improve the performance of
semi-supervised learning. However, we notice that, unlike general
classification tasks, in PbRL there exits a unique phenomenon that we defined
as similarity trap in this paper. Intuitively, human can have diametrically
opposite preferredness for similar segment pairs, but such similarity may trap
consistency regularization fail in PbRL. Due to the existence of similarity
trap, such consistency regularization improperly enhances the consistency
possiblity of the model's predictions between segment pairs, and thus reduces
the confidence in reward learning, since the augmented distribution does not
match with the original one in PbRL. To overcome such issue, we present a
self-training method along with our proposed peer regularization, which
penalizes the reward model memorizing uninformative labels and acquires
confident predictions. Empirically, we demonstrate that our approach is capable
of learning well a variety of locomotion and robotic manipulation behaviors
using different semi-supervised alternatives and peer regularization.