CR3DT: Camera-RADAR Fusion for 3D Detection and Tracking
Nicolas Baumann Michael Baumgartner Edoardo Ghignone Jonas Kühne Tobias Fischer Yung-Hsu Yang Marc Pollefeys Michele Magno
Accurate detection and tracking of surrounding objects is essential to enable
self-driving vehicles. While Light Detection and Ranging (LiDAR) sensors have
set the benchmark for high performance, the appeal of camera-only solutions
lies in their cost-effectiveness. Notably, despite the prevalent use of Radio
Detection and Ranging (RADAR) sensors in automotive systems, their potential in
3D detection and tracking has been largely disregarded due to data sparsity and
measurement noise. As a recent development, the combination of RADARs and
cameras is emerging as a promising solution. This paper presents Camera-RADAR
3D Detection and Tracking (CR3DT), a camera-RADAR fusion model for 3D object
detection, and Multi-Object Tracking (MOT). Building upon the foundations of
the State-of-the-Art (SotA) camera-only BEVDet architecture, CR3DT demonstrates
substantial improvements in both detection and tracking capabilities, by
incorporating the spatial and velocity information of the RADAR sensor.
Experimental results demonstrate an absolute improvement in detection
performance of 5.3% in mean Average Precision (mAP) and a 14.9% increase in
Average Multi-Object Tracking Accuracy (AMOTA) on the nuScenes dataset when
leveraging both modalities. CR3DT bridges the gap between high-performance and
cost-effective perception systems in autonomous driving, by capitalizing on the
ubiquitous presence of RADAR in automotive applications.