ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture
  Synthesis
Muhammad Hamza Mughal Rishabh Dabral Ikhsanul Habibie Lucia Donatelli Marc Habermann Christian Theobalt
Gestures play a key role in human communication. Recent methods for co-speech
gesture generation, while managing to generate beat-aligned motions, struggle
generating gestures that are semantically aligned with the utterance. Compared
to beat gestures that align naturally to the audio signal, semantically
coherent gestures require modeling the complex interactions between the
language and human motion, and can be controlled by focusing on certain words.
Therefore, we present ConvoFusion, a diffusion-based approach for multi-modal
gesture synthesis, which can not only generate gestures based on multi-modal
speech inputs, but can also facilitate controllability in gesture synthesis.
Our method proposes two guidance objectives that allow the users to modulate
the impact of different conditioning modalities (e.g. audio vs text) as well as
to choose certain words to be emphasized during gesturing. Our method is
versatile in that it can be trained either for generating monologue gestures or
even the conversational gestures. To further advance the research on
multi-party interactive gestures, the DnD Group Gesture dataset is released,
which contains 6 hours of gesture data showing 5 people interacting with one
another. We compare our method with several recent works and demonstrate
effectiveness of our method on a variety of tasks. We urge the reader to watch
our supplementary video at our website.