A Framework for Partially Observed Reward-States in RLHF
Chinmaya Kausik Mirco Mutti Aldo Pacchiano Ambuj Tewari
The study of reinforcement learning from human feedback (RLHF) has gained
prominence in recent years due to its role in the development of LLMs.
Neuroscience research shows that human responses to stimuli are known to depend
on partially-observed "internal states." Unfortunately current models of RLHF
do not take take this into consideration. Moreover most RLHF models do not
account for intermediate feedback, which is gaining importance in empirical
work and can help improve both sample complexity and alignment. To address
these limitations, we model RLHF as reinforcement learning with partially
observed reward-states (PORRL). We show reductions from the the two dominant
forms of human feedback in RLHF - cardinal and dueling feedback to PORRL. For
cardinal feedback, we develop generic statistically efficient algorithms and
instantiate them to present POR-UCRL and POR-UCBVI. For dueling feedback, we
show that a naive reduction to cardinal feedback fails to achieve sublinear
dueling regret. We then present the first explicit reduction that converts
guarantees for cardinal regret to dueling regret. We show that our models and
guarantees in both settings generalize and extend existing ones. Finally, we
identify a recursive structure on our model that could improve the statistical
and computational tractability of PORRL, giving examples from past work on RLHF
as well as learning perfect reward machines, which PORRL subsumes.