FitDiff: Robust monocular 3D facial shape and reflectance estimation
  using Diffusion Models
Stathis Galanakis Alexandros Lattas Stylianos Moschoglou Stefanos Zafeiriou
The remarkable progress in 3D face reconstruction has resulted in high-detail
and photorealistic facial representations. Recently, Diffusion Models have
revolutionized the capabilities of generative methods by achieving far better
performance than GANs. In this work, we present FitDiff, a diffusion-based 3D
facial avatar generative model. This model accurately generates relightable
facial avatars, utilizing an identity embedding extracted from an "in-the-wild"
2D facial image. Our multi-modal diffusion model concurrently outputs facial
reflectance maps (diffuse and specular albedo and normals) and shapes,
showcasing great generalization capabilities. It is solely trained on an
annotated subset of a public facial dataset, paired with 3D reconstructions. We
revisit the typical 3D facial fitting approach by guiding a reverse diffusion
process using perceptual and face recognition losses. Being the first LDM
conditioned on face recognition embeddings, FitDiff reconstructs relightable
human avatars, that can be used as-is in common rendering engines, starting
only from an unconstrained facial image, and achieving state-of-the-art
performance.