CorrTalk: Correlation Between Hierarchical Speech and Facial Activity
  Variances for 3D Animation
Zhaojie Chu Kailing Guo Xiaofen Xing Yilin Lan Bolun Cai Xiangmin Xu
Speech-driven 3D facial animation is a challenging cross-modal task that has
attracted growing research interest. During speaking activities, the mouth
displays strong motions, while the other facial regions typically demonstrate
comparatively weak activity levels. Existing approaches often simplify the
process by directly mapping single-level speech features to the entire facial
animation, which overlook the differences in facial activity intensity leading
to overly smoothed facial movements. In this study, we propose a novel
framework, CorrTalk, which effectively establishes the temporal correlation
between hierarchical speech features and facial activities of different
intensities across distinct regions. A novel facial activity intensity metric
is defined to distinguish between strong and weak facial activity, obtained by
computing the short-time Fourier transform of facial vertex displacements.
Based on the variances in facial activity, we propose a dual-branch decoding
framework to synchronously synthesize strong and weak facial activity, which
guarantees wider intensity facial animation synthesis. Furthermore, a weighted
hierarchical feature encoder is proposed to establish temporal correlation
between hierarchical speech features and facial activity at different
intensities, which ensures lip-sync and plausible facial expressions. Extensive
qualitatively and quantitatively experiments as well as a user study indicate
that our CorrTalk outperforms existing state-of-the-art methods. The source
code and supplementary video are publicly available at:
https://zjchu.github.io/projects/CorrTalk/