3D-Aware Talking-Head Video Motion Transfer
Haomiao Ni Jiachen Liu Yuan Xue Sharon X. Huang
Motion transfer of talking-head videos involves generating a new video with
the appearance of a subject video and the motion pattern of a driving video.
Current methodologies primarily depend on a limited number of subject images
and 2D representations, thereby neglecting to fully utilize the multi-view
appearance features inherent in the subject video. In this paper, we propose a
novel 3D-aware talking-head video motion transfer network, Head3D, which fully
exploits the subject appearance information by generating a
visually-interpretable 3D canonical head from the 2D subject frames with a
recurrent network. A key component of our approach is a self-supervised 3D head
geometry learning module, designed to predict head poses and depth maps from 2D
subject video frames. This module facilitates the estimation of a 3D head in
canonical space, which can then be transformed to align with driving video
frames. Additionally, we employ an attention-based fusion network to combine
the background and other details from subject frames with the 3D subject head
to produce the synthetic target video. Our extensive experiments on two public
talking-head video datasets demonstrate that Head3D outperforms both 2D and 3D
prior arts in the practical cross-identity setting, with evidence showing it
can be readily adapted to the pose-controllable novel view synthesis task.