Holo-Relighting: Controllable Volumetric Portrait Relighting from a
  Single Image
Yiqun Mei Yu Zeng He Zhang Zhixin Shu Xuaner Zhang Sai Bi Jianming Zhang HyunJoon Jung Vishal M. Patel
At the core of portrait photography is the search for ideal lighting and
viewpoint. The process often requires advanced knowledge in photography and an
elaborate studio setup. In this work, we propose Holo-Relighting, a volumetric
relighting method that is capable of synthesizing novel viewpoints, and novel
lighting from a single image. Holo-Relighting leverages the pretrained 3D GAN
(EG3D) to reconstruct geometry and appearance from an input portrait as a set
of 3D-aware features. We design a relighting module conditioned on a given
lighting to process these features, and predict a relit 3D representation in
the form of a tri-plane, which can render to an arbitrary viewpoint through
volume rendering. Besides viewpoint and lighting control, Holo-Relighting also
takes the head pose as a condition to enable head-pose-dependent lighting
effects. With these novel designs, Holo-Relighting can generate complex
non-Lambertian lighting effects (e.g., specular highlights and cast shadows)
without using any explicit physical lighting priors. We train Holo-Relighting
with data captured with a light stage, and propose two data-rendering
techniques to improve the data quality for training the volumetric relighting
system. Through quantitative and qualitative experiments, we demonstrate
Holo-Relighting can achieve state-of-the-arts relighting quality with better
photorealism, 3D consistency and controllability.