Direct Preference Optimization with an Offset
Afra Amini Tim Vieira Ryan Cotterell
Direct preference optimization (DPO) is a successful fine-tuning strategy for
aligning large language models with human preferences without the need to train
a reward model or employ reinforcement learning. DPO, as originally formulated,
relies on binary preference data and fine-tunes a language model to increase
the likelihood of a preferred response over a dispreferred response. However,
not all preference pairs are equal: while in some cases the preferred response
is only slightly better than the dispreferred response, there can be a stronger
preference for one response when, for example, the other response includes
harmful or toxic content. In this paper, we propose a generalization of DPO,
termed DPO with an offset (ODPO), that does not treat every preference pair
equally during fine-tuning. Intuitively, ODPO requires the difference between
the likelihood of the preferred and dispreferred response to be greater than an
offset value. The offset is determined based on the extent to which one
response is preferred over another. Our experiments on various tasks suggest
that ODPO significantly outperforms DPO in aligning language models, especially
when the number of preference pairs is limited.