Preference-free Alignment Learning with Regularized Relevance Reward
Sungdong Kim Minjoon Seo
Learning from human preference has been considered key to aligning Large
Language Models (LLMs) with human values. However, contrary to popular belief,
our preliminary study reveals that reward models trained on human preference
datasets tend to give higher scores to long off-topic responses than short
on-topic ones. Motivated by this observation, we explore a preference-free
approach utilizing `relevance' as a key objective for alignment. On our first
attempt, we find that the relevance score obtained by a retriever alone is
vulnerable to reward hacking, i.e., overoptimizing to undesired shortcuts, when
we utilize the score as a reward for reinforcement learning. To mitigate it, we
integrate effective inductive biases into the vanilla relevance to regularize
each other, resulting in a mixture of reward functions: Regularized Relevance
Reward ($R^3$). $R^3$ significantly improves performance on preference
benchmarks by providing a robust reward signal. Notably, $R^3$ does not require
any human preference datasets (i.e., preference-free), outperforming
open-source reward models in improving human preference. Our analysis
demonstrates that $R^3$ has advantages in elevating human preference while
minimizing its side effects. Finally, we show the generalizability of $R^3$,
consistently improving instruction-tuned models in various backbones and sizes
without additional dataset cost. Our code is available at
https://github.com/naver-ai/RRR.