Versatile Face Animator: Driving Arbitrary 3D Facial Avatar in RGBD
  Space
Haoyu Wang Haozhe Wu Junliang Xing Jia Jia
Creating realistic 3D facial animation is crucial for various applications in
the movie production and gaming industry, especially with the burgeoning demand
in the metaverse. However, prevalent methods such as blendshape-based
approaches and facial rigging techniques are time-consuming, labor-intensive,
and lack standardized configurations, making facial animation production
challenging and costly. In this paper, we propose a novel self-supervised
framework, Versatile Face Animator, which combines facial motion capture with
motion retargeting in an end-to-end manner, eliminating the need for
blendshapes or rigs. Our method has the following two main characteristics: 1)
we propose an RGBD animation module to learn facial motion from raw RGBD videos
by hierarchical motion dictionaries and animate RGBD images rendered from 3D
facial mesh coarse-to-fine, enabling facial animation on arbitrary 3D
characters regardless of their topology, textures, blendshapes, and rigs; and
2) we introduce a mesh retarget module to utilize RGBD animation to create 3D
facial animation by manipulating facial mesh with controller transformations,
which are estimated from dense optical flow fields and blended together with
geodesic-distance-based weights. Comprehensive experiments demonstrate the
effectiveness of our proposed framework in generating impressive 3D facial
animation results, highlighting its potential as a promising solution for the
cost-effective and efficient production of facial animation in the metaverse.