QPGesture: Quantization-Based and Phase-Guided Motion Matching for
  Natural Speech-Driven Gesture Generation
Sicheng Yang Zhiyong Wu Minglei Li Zhensong Zhang Lei Hao Weihong Bao Haolin Zhuang
Speech-driven gesture generation is highly challenging due to the random
jitters of human motion. In addition, there is an inherent asynchronous
relationship between human speech and gestures. To tackle these challenges, we
introduce a novel quantization-based and phase-guided motion-matching
framework. Specifically, we first present a gesture VQ-VAE module to learn a
codebook to summarize meaningful gesture units. With each code representing a
unique gesture, random jittering problems are alleviated effectively. We then
use Levenshtein distance to align diverse gestures with different speech.
Levenshtein distance based on audio quantization as a similarity metric of
corresponding speech of gestures helps match more appropriate gestures with
speech, and solves the alignment problem of speech and gestures well. Moreover,
we introduce phase to guide the optimal gesture matching based on the semantics
of context or rhythm of audio. Phase guides when text-based or speech-based
gestures should be performed to make the generated gestures more natural.
Extensive experiments show that our method outperforms recent approaches on
speech-driven gesture generation. Our code, database, pre-trained models, and
demos are available at https://github.com/YoungSeng/QPGesture.