IntrinsicAnything: Learning Diffusion Priors for Inverse Rendering Under
  Unknown Illumination
Xi Chen Sida Peng Dongchen Yang Yuan Liu Bowen Pan Chengfei Lv Xiaowei Zhou
This paper aims to recover object materials from posed images captured under
an unknown static lighting condition. Recent methods solve this task by
optimizing material parameters through differentiable physically based
rendering. However, due to the coupling between object geometry, materials, and
environment lighting, there is inherent ambiguity during the inverse rendering
process, preventing previous methods from obtaining accurate results. To
overcome this ill-posed problem, our key idea is to learn the material prior
with a generative model for regularizing the optimization process. We observe
that the general rendering equation can be split into diffuse and specular
shading terms, and thus formulate the material prior as diffusion models of
albedo and specular. Thanks to this design, our model can be trained using the
existing abundant 3D object data, and naturally acts as a versatile tool to
resolve the ambiguity when recovering material representations from RGB images.
In addition, we develop a coarse-to-fine training strategy that leverages
estimated materials to guide diffusion models to satisfy multi-view consistent
constraints, leading to more stable and accurate results. Extensive experiments
on real-world and synthetic datasets demonstrate that our approach achieves
state-of-the-art performance on material recovery. The code will be available
at https://zju3dv.github.io/IntrinsicAnything.