AE-NeRF: Audio Enhanced Neural Radiance Field for Few Shot Talking Head
  Synthesis
Dongze Li Kang Zhao Wei Wang Bo Peng Yingya Zhang Jing Dong Tieniu Tan
Audio-driven talking head synthesis is a promising topic with wide
applications in digital human, film making and virtual reality. Recent
NeRF-based approaches have shown superiority in quality and fidelity compared
to previous studies. However, when it comes to few-shot talking head
generation, a practical scenario where only few seconds of talking video is
available for one identity, two limitations emerge: 1) they either have no base
model, which serves as a facial prior for fast convergence, or ignore the
importance of audio when building the prior; 2) most of them overlook the
degree of correlation between different face regions and audio, e.g., mouth is
audio related, while ear is audio independent. In this paper, we present Audio
Enhanced Neural Radiance Field (AE-NeRF) to tackle the above issues, which can
generate realistic portraits of a new speaker with fewshot dataset.
Specifically, we introduce an Audio Aware Aggregation module into the feature
fusion stage of the reference scheme, where the weight is determined by the
similarity of audio between reference and target image. Then, an Audio-Aligned
Face Generation strategy is proposed to model the audio related and audio
independent regions respectively, with a dual-NeRF framework. Extensive
experiments have shown AE-NeRF surpasses the state-of-the-art on image
fidelity, audio-lip synchronization, and generalization ability, even in
limited training set or training iterations.