Mixed Preference Optimization: Reinforcement Learning with Data
  Selection and Better Reference Model
Qi Gou Cam-Tu Nguyen
Large Language Models (LLMs) have become increasingly popular due to their
ability to process and generate natural language. However, as they are trained
on massive datasets of text, LLMs can inherit harmful biases and produce
outputs that are not aligned with human values. This paper studies two main
approaches to LLM alignment: Reinforcement Learning with Human Feedback (RLHF)
and contrastive learning-based methods like Direct Preference Optimization
(DPO). By analyzing the stability and robustness of RLHF and DPO, we propose
MPO (Mixed Preference Optimization), a novel method that mitigates the
weaknesses of both approaches. Specifically, we propose a two-stage training
procedure: first train DPO on an easy dataset, and then perform RLHF on a
difficult set with DPO model being the reference model. Here, the easy and
difficult sets are constructed by a well-trained reward model that splits
response pairs into those with large gaps of reward (easy), and those with
small gaps (difficult). The first stage allows us to obtain a relatively
optimal policy (LLM) model quickly, whereas the second stage refines LLM with
online RLHF, thus mitigating the distribution shift issue associated with DPO.
Experiments are conducted on two public alignment datasets, namely HH-RLHF and
TLDR, demonstrating the effectiveness of MPO, both in terms of GPT4 and human
evaluation.