mmFUSION: Multimodal Fusion for 3D Objects Detection
Javed Ahmad Alessio Del Bue
Multi-sensor fusion is essential for accurate 3D object detection in
self-driving systems. Camera and LiDAR are the most commonly used sensors, and
usually, their fusion happens at the early or late stages of 3D detectors with
the help of regions of interest (RoIs). On the other hand, fusion at the
intermediate level is more adaptive because it does not need RoIs from
modalities but is complex as the features of both modalities are presented from
different points of view. In this paper, we propose a new intermediate-level
multi-modal fusion (mmFUSION) approach to overcome these challenges. First, the
mmFUSION uses separate encoders for each modality to compute features at a
desired lower space volume. Second, these features are fused through
cross-modality and multi-modality attention mechanisms proposed in mmFUSION.
The mmFUSION framework preserves multi-modal information and learns to
complement modalities' deficiencies through attention weights. The strong
multi-modal features from the mmFUSION framework are fed to a simple 3D
detection head for 3D predictions. We evaluate mmFUSION on the KITTI and
NuScenes dataset where it performs better than available early, intermediate,
late, and even two-stage based fusion schemes. The code with the mmdetection3D
project plugin will be publicly available soon.