MAGMA: Music Aligned Generative Motion Autodecoder
Sohan Anisetty Amit Raj James Hays
Mapping music to dance is a challenging problem that requires spatial and
temporal coherence along with a continual synchronization with the music's
progression. Taking inspiration from large language models, we introduce a
2-step approach for generating dance using a Vector Quantized-Variational
Autoencoder (VQ-VAE) to distill motion into primitives and train a Transformer
decoder to learn the correct sequencing of these primitives. We also evaluate
the importance of music representations by comparing naive music feature
extraction using Librosa to deep audio representations generated by
state-of-the-art audio compression algorithms. Additionally, we train
variations of the motion generator using relative and absolute positional
encodings to determine the effect on generated motion quality when generating
arbitrarily long sequence lengths. Our proposed approach achieve
state-of-the-art results in music-to-motion generation benchmarks and enables
the real-time generation of considerably longer motion sequences, the ability
to chain multiple motion sequences seamlessly, and easy customization of motion
sequences to meet style requirements.