TCLC-GS: Tightly Coupled LiDAR-Camera Gaussian Splatting for Surrounding
  Autonomous Driving Scenes
Cheng Zhao Su Sun Ruoyu Wang Yuliang Guo Jun-Jun Wan Zhou Huang Xinyu Huang Yingjie Victor Chen Liu Ren
Most 3D Gaussian Splatting (3D-GS) based methods for urban scenes initialize
3D Gaussians directly with 3D LiDAR points, which not only underutilizes LiDAR
data capabilities but also overlooks the potential advantages of fusing LiDAR
with camera data. In this paper, we design a novel tightly coupled LiDAR-Camera
Gaussian Splatting (TCLC-GS) to fully leverage the combined strengths of both
LiDAR and camera sensors, enabling rapid, high-quality 3D reconstruction and
novel view RGB/depth synthesis. TCLC-GS designs a hybrid explicit (colorized 3D
mesh) and implicit (hierarchical octree feature) 3D representation derived from
LiDAR-camera data, to enrich the properties of 3D Gaussians for splatting. 3D
Gaussian's properties are not only initialized in alignment with the 3D mesh
which provides more completed 3D shape and color information, but are also
endowed with broader contextual information through retrieved octree implicit
features. During the Gaussian Splatting optimization process, the 3D mesh
offers dense depth information as supervision, which enhances the training
process by learning of a robust geometry. Comprehensive evaluations conducted
on the Waymo Open Dataset and nuScenes Dataset validate our method's
state-of-the-art (SOTA) performance. Utilizing a single NVIDIA RTX 3090 Ti, our
method demonstrates fast training and achieves real-time RGB and depth
rendering at 90 FPS in resolution of 1920x1280 (Waymo), and 120 FPS in
resolution of 1600x900 (nuScenes) in urban scenarios.