MusicRL: Aligning Music Generation to Human Preferences
Geoffrey Cideron Sertan Girgin Mauro Verzetti Damien Vincent Matej Kastelic Zalán Borsos Brian McWilliams Victor Ungureanu Olivier Bachem Olivier Pietquin Matthieu Geist Léonard Hussenot Neil Zeghidour Andrea Agostinelli
We propose MusicRL, the first music generation system finetuned from human
feedback. Appreciation of text-to-music models is particularly subjective since
the concept of musicality as well as the specific intention behind a caption
are user-dependent (e.g. a caption such as "upbeat work-out music" can map to a
retro guitar solo or a techno pop beat). Not only this makes supervised
training of such models challenging, but it also calls for integrating
continuous human feedback in their post-deployment finetuning. MusicRL is a
pretrained autoregressive MusicLM (Agostinelli et al., 2023) model of discrete
audio tokens finetuned with reinforcement learning to maximise sequence-level
rewards. We design reward functions related specifically to text-adherence and
audio quality with the help from selected raters, and use those to finetune
MusicLM into MusicRL-R. We deploy MusicLM to users and collect a substantial
dataset comprising 300,000 pairwise preferences. Using Reinforcement Learning
from Human Feedback (RLHF), we train MusicRL-U, the first text-to-music model
that incorporates human feedback at scale. Human evaluations show that both
MusicRL-R and MusicRL-U are preferred to the baseline. Ultimately, MusicRL-RU
combines the two approaches and results in the best model according to human
raters. Ablation studies shed light on the musical attributes influencing human
preferences, indicating that text adherence and quality only account for a part
of it. This underscores the prevalence of subjectivity in musical appreciation
and calls for further involvement of human listeners in the finetuning of music
generation models.