Active Preference Learning for Large Language Models
William Muldrew Peter Hayes Mingtian Zhang David Barber
As large language models (LLMs) become more capable, fine-tuning techniques
for aligning with human intent are increasingly important. A key consideration
for aligning these models is how to most effectively use human resources, or
model resources in the case where LLMs themselves are used as oracles.
Reinforcement learning from Human or AI preferences (RLHF/RLAIF) is the most
prominent example of such a technique, but is complex and often unstable.
Direct Preference Optimization (DPO) has recently been proposed as a simpler
and more stable alternative. In this work, we develop an active learning
strategy for DPO to make better use of preference labels. We propose a
practical acquisition function for prompt/completion pairs based on the
predictive entropy of the language model and a measure of certainty of the
implicit preference model optimized by DPO. We demonstrate how our approach
improves both the rate of learning and final performance of fine-tuning on
pairwise preference data.