Music- and Lyrics-driven Dance Synthesis
Wenjie Yin Qingyuan Yao Yi Yu Hang Yin Danica Kragic Mårten Björkman
Lyrics often convey information about the songs that are beyond the auditory
dimension, enriching the semantic meaning of movements and musical themes. Such
insights are important in the dance choreography domain. However, most existing
dance synthesis methods mainly focus on music-to-dance generation, without
considering the semantic information. To complement it, we introduce JustLMD, a
new multimodal dataset of 3D dance motion with music and lyrics. To the best of
our knowledge, this is the first dataset with triplet information including
dance motion, music, and lyrics. Additionally, we showcase a cross-modal
diffusion-based network designed to generate 3D dance motion conditioned on
music and lyrics. The proposed JustLMD dataset encompasses 4.6 hours of 3D
dance motion in 1867 sequences, accompanied by musical tracks and their
corresponding English lyrics.