Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering
Linjie Lyu Ayush Tewari Marc Habermann Shunsuke Saito Michael Zollhöfer Thomas Leimkühler Christian Theobalt
Inverse rendering, the process of inferring scene properties from images, is
a challenging inverse problem. The task is ill-posed, as many different scene
configurations can give rise to the same image. Most existing solutions
incorporate priors into the inverse-rendering pipeline to encourage plausible
solutions, but they do not consider the inherent ambiguities and the
multi-modal distribution of possible decompositions. In this work, we propose a
novel scheme that integrates a denoising diffusion probabilistic model
pre-trained on natural illumination maps into an optimization framework
involving a differentiable path tracer. The proposed method allows sampling
from combinations of illumination and spatially-varying surface materials that
are, both, natural and explain the image observations. We further conduct an
extensive comparative study of different priors on illumination used in
previous work on inverse rendering. Our method excels in recovering materials
and producing highly realistic and diverse environment map samples that
faithfully explain the illumination of the input images.