Portrait4D-v2: Pseudo Multi-View Data Creates Better 4D Head Synthesizer
Yu Deng Duomin Wang Baoyuan Wang
In this paper, we propose a novel learning approach for feed-forward one-shot
4D head avatar synthesis. Different from existing methods that often learn from
reconstructing monocular videos guided by 3DMM, we employ pseudo multi-view
videos to learn a 4D head synthesizer in a data-driven manner, avoiding
reliance on inaccurate 3DMM reconstruction that could be detrimental to the
synthesis performance. The key idea is to first learn a 3D head synthesizer
using synthetic multi-view images to convert monocular real videos into
multi-view ones, and then utilize the pseudo multi-view videos to learn a 4D
head synthesizer via cross-view self-reenactment. By leveraging a simple vision
transformer backbone with motion-aware cross-attentions, our method exhibits
superior performance compared to previous methods in terms of reconstruction
fidelity, geometry consistency, and motion control accuracy. We hope our method
offers novel insights into integrating 3D priors with 2D supervisions for
improved 4D head avatar creation.