The Past, Present and Better Future of Feedback Learning in Large
  Language Models for Subjective Human Preferences and Values
Hannah Rose Kirk Andrew M. Bean Bertie Vidgen Paul Röttger Scott A. Hale
Human feedback is increasingly used to steer the behaviours of Large Language
Models (LLMs). However, it is unclear how to collect and incorporate feedback
in a way that is efficient, effective and unbiased, especially for highly
subjective human preferences and values. In this paper, we survey existing
approaches for learning from human feedback, drawing on 95 papers primarily
from the ACL and arXiv repositories.First, we summarise the past, pre-LLM
trends for integrating human feedback into language models. Second, we give an
overview of present techniques and practices, as well as the motivations for
using feedback; conceptual frameworks for defining values and preferences; and
how feedback is collected and from whom. Finally, we encourage a better future
of feedback learning in LLMs by raising five unresolved conceptual and
practical challenges.