A Survey of Reinforcement Learning from Human Feedback
Timo Kaufmann Paul Weng Viktor Bengs Eyke Hüllermeier
Reinforcement learning from human feedback (RLHF) is a variant of
reinforcement learning (RL) that learns from human feedback instead of relying
on an engineered reward function. Building on prior work on the related setting
of preference-based reinforcement learning (PbRL), it stands at the
intersection of artificial intelligence and human-computer interaction. This
positioning offers a promising avenue to enhance the performance and
adaptability of intelligent systems while also improving the alignment of their
objectives with human values. The training of Large Language Models (LLMs) has
impressively demonstrated this potential in recent years, where RLHF played a
decisive role in targeting the model's capabilities toward human objectives.
This article provides a comprehensive overview of the fundamentals of RLHF,
exploring the intricate dynamics between machine agents and human input. While
recent focus has been on RLHF for LLMs, our survey adopts a broader
perspective, examining the diverse applications and wide-ranging impact of the
technique. We delve into the core principles that underpin RLHF, shedding light
on the symbiotic relationship between algorithms and human feedback, and
discuss the main research trends in the field. By synthesizing the current
landscape of RLHF research, this article aims to provide researchers as well as
practitioners with a comprehensive understanding of this rapidly growing field
of research.