Sample-Efficient Preference-based Reinforcement Learning with Dynamics
  Aware Rewards
Katherine Metcalf Miguel Sarabia Natalie Mackraz Barry-John Theobald
Preference-based reinforcement learning (PbRL) aligns a robot behavior with
human preferences via a reward function learned from binary feedback over agent
behaviors. We show that dynamics-aware reward functions improve the sample
efficiency of PbRL by an order of magnitude. In our experiments we iterate
between: (1) learning a dynamics-aware state-action representation (z^{sa}) via
a self-supervised temporal consistency task, and (2) bootstrapping the
preference-based reward function from (z^{sa}), which results in faster policy
learning and better final policy performance. For example, on quadruped-walk,
walker-walk, and cheetah-run, with 50 preference labels we achieve the same
performance as existing approaches with 500 preference labels, and we recover
83\% and 66\% of ground truth reward policy performance versus only 38\% and
21\%. The performance gains demonstrate the benefits of explicitly learning a
dynamics-aware reward model. Repo: \texttt{https://github.com/apple/ml-reed}.