Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance
  Accompaniment
Li Siyao Tianpei Gu Zhitao Yang Zhengyu Lin Ziwei Liu Henghui Ding Lei Yang Chen Change Loy
We introduce a novel task within the field of 3D dance generation, termed
dance accompaniment, which necessitates the generation of responsive movements
from a dance partner, the "follower", synchronized with the lead dancer's
movements and the underlying musical rhythm. Unlike existing solo or group
dance generation tasks, a duet dance scenario entails a heightened degree of
interaction between the two participants, requiring delicate coordination in
both pose and position. To support this task, we first build a large-scale and
diverse duet interactive dance dataset, DD100, by recording about 117 minutes
of professional dancers' performances. To address the challenges inherent in
this task, we propose a GPT-based model, Duolando, which autoregressively
predicts the subsequent tokenized motion conditioned on the coordinated
information of the music, the leader's and the follower's movements. To further
enhance the GPT's capabilities of generating stable results on unseen
conditions (music and leader motions), we devise an off-policy reinforcement
learning strategy that allows the model to explore viable trajectories from
out-of-distribution samplings, guided by human-defined rewards. Based on the
collected dataset and proposed method, we establish a benchmark with several
carefully designed metrics.