On the Essence and Prospect: An Investigation of Alignment Approaches
  for Big Models
Xinpeng Wang Shitong Duan Xiaoyuan Yi Jing Yao Shanlin Zhou Zhihua Wei Peng Zhang Dongkuan Xu Maosong Sun Xing Xie
Big models have achieved revolutionary breakthroughs in the field of AI, but
they might also pose potential concerns. Addressing such concerns, alignment
technologies were introduced to make these models conform to human preferences
and values. Despite considerable advancements in the past year, various
challenges lie in establishing the optimal alignment strategy, such as data
cost and scalable oversight, and how to align remains an open question. In this
survey paper, we comprehensively investigate value alignment approaches. We
first unpack the historical context of alignment tracing back to the 1920s
(where it comes from), then delve into the mathematical essence of alignment
(what it is), shedding light on the inherent challenges. Following this
foundation, we provide a detailed examination of existing alignment methods,
which fall into three categories: Reinforcement Learning, Supervised
Fine-Tuning, and In-context Learning, and demonstrate their intrinsic
connections, strengths, and limitations, helping readers better understand this
research area. In addition, two emerging topics, personal alignment, and
multimodal alignment, are also discussed as novel frontiers in this field.
Looking forward, we discuss potential alignment paradigms and how they could
handle remaining challenges, prospecting where future alignment will go.