Chain of Generation: Multi-Modal Gesture Synthesis via Cascaded
  Conditional Control
Zunnan Xu Yachao Zhang Sicheng Yang Ronghui Li Xiu Li
This study aims to improve the generation of 3D gestures by utilizing
multimodal information from human speech. Previous studies have focused on
incorporating additional modalities to enhance the quality of generated
gestures. However, these methods perform poorly when certain modalities are
missing during inference. To address this problem, we suggest using
speech-derived multimodal priors to improve gesture generation. We introduce a
novel method that separates priors from speech and employs multimodal priors as
constraints for generating gestures. Our approach utilizes a chain-like
modeling method to generate facial blendshapes, body movements, and hand
gestures sequentially. Specifically, we incorporate rhythm cues derived from
facial deformation and stylization prior based on speech emotions, into the
process of generating gestures. By incorporating multimodal priors, our method
improves the quality of generated gestures and eliminate the need for expensive
setup preparation during inference. Extensive experiments and user studies
confirm that our proposed approach achieves state-of-the-art performance.