DifFUSER: Diffusion Model for Robust Multi-Sensor Fusion in 3D Object
  Detection and BEV Segmentation
Duy-Tho Le Hengcan Shi Jianfei Cai Hamid Rezatofighi
Diffusion models have recently gained prominence as powerful deep generative
models, demonstrating unmatched performance across various domains. However,
their potential in multi-sensor fusion remains largely unexplored. In this
work, we introduce DifFUSER, a novel approach that leverages diffusion models
for multi-modal fusion in 3D object detection and BEV map segmentation.
Benefiting from the inherent denoising property of diffusion, DifFUSER is able
to refine or even synthesize sensor features in case of sensor malfunction,
thereby improving the quality of the fused output. In terms of architecture,
our DifFUSER blocks are chained together in a hierarchical BiFPN fashion,
termed cMini-BiFPN, offering an alternative architecture for latent diffusion.
We further introduce a Gated Self-conditioned Modulated (GSM) latent diffusion
module together with a Progressive Sensor Dropout Training (PSDT) paradigm,
designed to add stronger conditioning to the diffusion process and robustness
to sensor failures. Our extensive evaluations on the Nuscenes dataset reveal
that DifFUSER not only achieves state-of-the-art performance with a 69.1% mIOU
in BEV map segmentation tasks but also competes effectively with leading
transformer-based fusion techniques in 3D object detection.