SpeechAlign: Aligning Speech Generation to Human Preferences
Dong Zhang Zhaowei Li Shimin Li Xin Zhang Pengyu Wang Yaqian Zhou Xipeng Qiu
Speech language models have significantly advanced in generating realistic
speech, with neural codec language models standing out. However, the
integration of human feedback to align speech outputs to human preferences is
often neglected. This paper addresses this gap by first analyzing the
distribution gap in codec language models, highlighting how it leads to
discrepancies between the training and inference phases, which negatively
affects performance. Then we explore leveraging learning from human feedback to
bridge the distribution gap. We introduce SpeechAlign, an iterative
self-improvement strategy that aligns speech language models to human
preferences. SpeechAlign involves constructing a preference codec dataset
contrasting golden codec tokens against synthetic tokens, followed by
preference optimization to improve the codec language model. This cycle of
improvement is carried out iteratively to steadily convert weak models to
strong ones. Through both subjective and objective evaluations, we show that
SpeechAlign can bridge the distribution gap and facilitating continuous
self-improvement of the speech language model. Moreover, SpeechAlign exhibits
robust generalization capabilities and works for smaller models. Code and
models will be available at https://github.com/0nutation/SpeechGPT.