Attention-Based VR Facial Animation with Visual Mouth Camera Guidance
  for Immersive Telepresence Avatars
Andre Rochow Max Schwarz Sven Behnke
Facial animation in virtual reality environments is essential for
applications that necessitate clear visibility of the user's face and the
ability to convey emotional signals. In our scenario, we animate the face of an
operator who controls a robotic Avatar system. The use of facial animation is
particularly valuable when the perception of interacting with a specific
individual, rather than just a robot, is intended. Purely keypoint-driven
animation approaches struggle with the complexity of facial movements. We
present a hybrid method that uses both keypoints and direct visual guidance
from a mouth camera. Our method generalizes to unseen operators and requires
only a quick enrolment step with capture of two short videos. Multiple source
images are selected with the intention to cover different facial expressions.
Given a mouth camera frame from the HMD, we dynamically construct the target
keypoints and apply an attention mechanism to determine the importance of each
source image. To resolve keypoint ambiguities and animate a broader range of
mouth expressions, we propose to inject visual mouth camera information into
the latent space. We enable training on large-scale speaking head datasets by
simulating the mouth camera input with its perspective differences and facial
deformations. Our method outperforms a baseline in quality, capability, and
temporal consistency. In addition, we highlight how the facial animation
contributed to our victory at the ANA Avatar XPRIZE Finals.