3D Face Style Transfer with a Hybrid Solution of NeRF and Mesh
  Rasterization
Jianwei Feng Prateek Singhal
Style transfer for human face has been widely researched in recent years.
Majority of the existing approaches work in 2D image domain and have 3D
inconsistency issue when applied on different viewpoints of the same face. In
this paper, we tackle the problem of 3D face style transfer which aims at
generating stylized novel views of a 3D human face with multi-view consistency.
We propose to use a neural radiance field (NeRF) to represent 3D human face and
combine it with 2D style transfer to stylize the 3D face. We find that directly
training a NeRF on stylized images from 2D style transfer brings in 3D
inconsistency issue and causes blurriness. On the other hand, training a NeRF
jointly with 2D style transfer objectives shows poor convergence due to the
identity and head pose gap between style image and content image. It also poses
challenge in training time and memory due to the need of volume rendering for
full image to apply style transfer loss functions. We therefore propose a
hybrid framework of NeRF and mesh rasterization to combine the benefits of high
fidelity geometry reconstruction of NeRF and fast rendering speed of mesh. Our
framework consists of three stages: 1. Training a NeRF model on input face
images to learn the 3D geometry; 2. Extracting a mesh from the trained NeRF
model and optimizing it with style transfer objectives via differentiable
rasterization; 3. Training a new color network in NeRF conditioned on a style
embedding to enable arbitrary style transfer to the 3D face. Experiment results
show that our approach generates high quality face style transfer with great 3D
consistency, while also enabling a flexible style control.