Inverse Rendering of Glossy Objects via the Neural Plenoptic Function
  and Radiance Fields
Haoyuan Wang Wenbo Hu Lei Zhu Rynson W. H. Lau
Inverse rendering aims at recovering both geometry and materials of objects.
It provides a more compatible reconstruction for conventional rendering
engines, compared with the neural radiance fields (NeRFs). On the other hand,
existing NeRF-based inverse rendering methods cannot handle glossy objects with
local light interactions well, as they typically oversimplify the illumination
as a 2D environmental map, which assumes infinite lights only. Observing the
superiority of NeRFs in recovering radiance fields, we propose a novel 5D
Neural Plenoptic Function (NeP) based on NeRFs and ray tracing, such that more
accurate lighting-object interactions can be formulated via the rendering
equation. We also design a material-aware cone sampling strategy to efficiently
integrate lights inside the BRDF lobes with the help of pre-filtered radiance
fields. Our method has two stages: the geometry of the target object and the
pre-filtered environmental radiance fields are reconstructed in the first
stage, and materials of the target object are estimated in the second stage
with the proposed NeP and material-aware cone sampling strategy. Extensive
experiments on the proposed real-world and synthetic datasets demonstrate that
our method can reconstruct high-fidelity geometry/materials of challenging
glossy objects with complex lighting interactions from nearby objects. Project
webpage: https://whyy.site/paper/nep