A Minimaximalist Approach to Reinforcement Learning from Human Feedback
Gokul Swamy Christoph Dann Rahul Kidambi Zhiwei Steven Wu Alekh Agarwal
We present Self-Play Preference Optimization (SPO), an algorithm for
reinforcement learning from human feedback. Our approach is minimalist in that
it does not require training a reward model nor unstable adversarial training
and is therefore rather simple to implement. Our approach is maximalist in that
it provably handles non-Markovian, intransitive, and stochastic preferences
while being robust to the compounding errors that plague offline approaches to
sequential prediction. To achieve the preceding qualities, we build upon the
concept of a Minimax Winner (MW), a notion of preference aggregation from the
social choice theory literature that frames learning from preferences as a
zero-sum game between two policies. By leveraging the symmetry of this game, we
prove that rather than using the traditional technique of dueling two policies
to compute the MW, we can simply have a single agent play against itself while
maintaining strong convergence guarantees. Practically, this corresponds to
sampling multiple trajectories from a policy, asking a rater or preference
model to compare them, and then using the proportion of wins as the reward for
a particular trajectory. We demonstrate that on a suite of continuous control
tasks, we are able to learn significantly more efficiently than reward-model
based approaches while maintaining robustness to the intransitive and
stochastic preferences that frequently occur in practice when aggregating human
judgments.