Parametric Implicit Face Representation for Audio-Driven Facial
  Reenactment
Ricong Huang Peiwen Lai Yipeng Qin Guanbin Li
Audio-driven facial reenactment is a crucial technique that has a range of
applications in film-making, virtual avatars and video conferences. Existing
works either employ explicit intermediate face representations (e.g., 2D facial
landmarks or 3D face models) or implicit ones (e.g., Neural Radiance Fields),
thus suffering from the trade-offs between interpretability and expressive
power, hence between controllability and quality of the results. In this work,
we break these trade-offs with our novel parametric implicit face
representation and propose a novel audio-driven facial reenactment framework
that is both controllable and can generate high-quality talking heads.
Specifically, our parametric implicit representation parameterizes the implicit
representation with interpretable parameters of 3D face models, thereby taking
the best of both explicit and implicit methods. In addition, we propose several
new techniques to improve the three components of our framework, including i)
incorporating contextual information into the audio-to-expression parameters
encoding; ii) using conditional image synthesis to parameterize the implicit
representation and implementing it with an innovative tri-plane structure for
efficient learning; iii) formulating facial reenactment as a conditional image
inpainting problem and proposing a novel data augmentation technique to improve
model generalizability. Extensive experiments demonstrate that our method can
generate more realistic results than previous methods with greater fidelity to
the identities and talking styles of speakers.