Binary Classifier Optimization for Large Language Model Alignment
Seungjae Jung Gunsoo Han Daniel Wontae Nam Kyoung-Woon On
Aligning Large Language Models (LLMs) to human preferences through preference
optimization has been crucial but labor-intensive, necessitating for each
prompt a comparison of both a chosen and a rejected text completion by
evaluators. Recently, Kahneman-Tversky Optimization (KTO) has demonstrated that
LLMs can be aligned using merely binary "thumbs-up" or "thumbs-down" signals on
each prompt-completion pair. In this paper, we present theoretical foundations
to explain the successful alignment achieved through these binary signals. Our
analysis uncovers a new perspective: optimizing a binary classifier, whose
logit is a reward, implicitly induces minimizing the Direct Preference
Optimization (DPO) loss. In the process of this discovery, we identified two
techniques for effective alignment: reward shift and underlying distribution
matching. Consequently, we propose a new algorithm, \textit{Binary Classifier
Optimization}, that integrates the techniques. We validate our methodology in
two settings: first, on a paired preference dataset, where our method performs
on par with DPO and KTO; and second, on binary signal datasets simulating
real-world conditions with divergent underlying distributions between thumbs-up
and thumbs-down data. Our model consistently demonstrates effective and robust
alignment across two base LLMs and three different binary signal datasets,
showcasing the strength of our approach to learning from binary feedback.