DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose
  Generation via Diffusion Models
Zhiyao Sun Tian Lv Sheng Ye Matthieu Gaetan Lin Jenny Sheng Yu-Hui Wen Minjing Yu Yong-jin Liu
The generation of stylistic 3D facial animations driven by speech poses a
significant challenge as it requires learning a many-to-many mapping between
speech, style, and the corresponding natural facial motion. However, existing
methods either employ a deterministic model for speech-to-motion mapping or
encode the style using a one-hot encoding scheme. Notably, the one-hot encoding
approach fails to capture the complexity of the style and thus limits
generalization ability. In this paper, we propose DiffPoseTalk, a generative
framework based on the diffusion model combined with a style encoder that
extracts style embeddings from short reference videos. During inference, we
employ classifier-free guidance to guide the generation process based on the
speech and style. We extend this to include the generation of head poses,
thereby enhancing user perception. Additionally, we address the shortage of
scanned 3D talking face data by training our model on reconstructed 3DMM
parameters from a high-quality, in-the-wild audio-visual dataset. Our extensive
experiments and user study demonstrate that our approach outperforms
state-of-the-art methods. The code and dataset will be made publicly available.