KTO: Model Alignment as Prospect Theoretic Optimization
Kawin Ethayarajh Winnie Xu Niklas Muennighoff Dan Jurafsky Douwe Kiela
Kahneman & Tversky's $\textit{prospect theory}$ tells us that humans perceive
random variables in a biased but well-defined manner; for example, humans are
famously loss-averse. We show that objectives for aligning LLMs with human
feedback implicitly incorporate many of these biases -- the success of these
objectives (e.g., DPO) over cross-entropy minimization can partly be ascribed
to them being $\textit{human-aware loss functions}$ (HALOs). However, the
utility functions these methods attribute to humans still differ from those in
the prospect theory literature. Using a Kahneman-Tversky model of human
utility, we propose a HALO that directly maximizes the utility of generations
instead of maximizing the log-likelihood of preferences, as current methods do.
We call this approach Kahneman-Tversky Optimization (KTO), and it matches or
exceeds the performance of preference-based methods at scales from 1B to 30B.
Crucially, KTO does not need preferences -- only a binary signal of whether an
output is desirable or undesirable for a given input. This makes it far easier
to use in the real world, where preference data is scarce and expensive.