Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF
  Decomposition and Ray Tracing
Jian Gao Chun Gu Youtian Lin Hao Zhu Xun Cao Li Zhang Yao Yao
We present a novel differentiable point-based rendering framework for
material and lighting decomposition from multi-view images, enabling editing,
ray-tracing, and real-time relighting of the 3D point cloud. Specifically, a 3D
scene is represented as a set of relightable 3D Gaussian points, where each
point is additionally associated with a normal direction, BRDF parameters, and
incident lights from different directions. To achieve robust lighting
estimation, we further divide incident lights of each point into global and
local components, as well as view-dependent visibilities. The 3D scene is
optimized through the 3D Gaussian Splatting technique while BRDF and lighting
are decomposed by physically-based differentiable rendering. Moreover, we
introduce an innovative point-based ray-tracing approach based on the bounding
volume hierarchy for efficient visibility baking, enabling real-time rendering
and relighting of 3D Gaussian points with accurate shadow effects. Extensive
experiments demonstrate improved BRDF estimation and novel view rendering
results compared to state-of-the-art material estimation approaches. Our
framework showcases the potential to revolutionize the mesh-based graphics
pipeline with a relightable, traceable, and editable rendering pipeline solely
based on point cloud. Project
page:https://nju-3dv.github.io/projects/Relightable3DGaussian/.