Noise Contrastive Alignment of Language Models with Explicit Rewards
Huayu Chen Guande He Hang Su Jun Zhu
User intentions are typically formalized as evaluation rewards to be
maximized when fine-tuning language models (LMs). Existing alignment methods,
such as Direct Preference Optimization (DPO), are mainly tailored for pairwise
preference data where rewards are implicitly defined rather than explicitly
given. In this paper, we introduce a general framework for LM alignment,
leveraging Noise Contrastive Estimation (NCE) to bridge the gap in handling
reward datasets explicitly annotated with scalar evaluations. Our framework
comprises two parallel algorithms, NCA and InfoNCA, both enabling the direct
extraction of an LM policy from reward data as well as preference data.
Notably, we show that the DPO loss is a special case of our proposed InfoNCA
objective under pairwise preference settings, thereby integrating and extending
current alignment theories. By contrasting NCA and InfoNCA, we show that
InfoNCA and DPO adjust relative likelihood across different responses to a
single instruction, while NCA optimizes absolute likelihood for each response.
We apply our methods to align a 7B language model with a GPT-4 annotated reward
dataset. Experimental results suggest that InfoNCA surpasses the DPO baseline
in GPT-4 evaluations, while NCA enjoys better training stability with
competitive performance.