Towards Realistic Scene Generation with LiDAR Diffusion Models
Haoxi Ran Vitor Guizilini Yue Wang
Diffusion models (DMs) excel in photo-realistic image synthesis, but their
adaptation to LiDAR scene generation poses a substantial hurdle. This is
primarily because DMs operating in the point space struggle to preserve the
curve-like patterns and 3D geometry of LiDAR scenes, which consumes much of
their representation power. In this paper, we propose LiDAR Diffusion Models
(LiDMs) to generate LiDAR-realistic scenes from a latent space tailored to
capture the realism of LiDAR scenes by incorporating geometric priors into the
learning pipeline. Our method targets three major desiderata: pattern realism,
geometry realism, and object realism. Specifically, we introduce curve-wise
compression to simulate real-world LiDAR patterns, point-wise coordinate
supervision to learn scene geometry, and patch-wise encoding for a full 3D
object context. With these three core designs, our method achieves competitive
performance on unconditional LiDAR generation in 64-beam scenario and state of
the art on conditional LiDAR generation, while maintaining high efficiency
compared to point-based DMs (up to 107$\times$ faster). Furthermore, by
compressing LiDAR scenes into a latent space, we enable the controllability of
DMs with various conditions such as semantic maps, camera views, and text
prompts. Our code and pretrained weights are available at
https://github.com/hancyran/LiDAR-Diffusion.