Multimodal Object Query Initialization for 3D Object Detection
Mathijs R. van Geerenstein Felicia Ruppel Klaus Dietmayer Dariu M. Gavrila
3D object detection models that exploit both LiDAR and camera sensor features
are top performers in large-scale autonomous driving benchmarks. A transformer
is a popular network architecture used for this task, in which so-called object
queries act as candidate objects. Initializing these object queries based on
current sensor inputs is a common practice. For this, existing methods strongly
rely on LiDAR data however, and do not fully exploit image features. Besides,
they introduce significant latency. To overcome these limitations we propose
EfficientQ3M, an efficient, modular, and multimodal solution for object query
initialization for transformer-based 3D object detection models. The proposed
initialization method is combined with a "modality-balanced" transformer
decoder where the queries can access all sensor modalities throughout the
decoder. In experiments, we outperform the state of the art in
transformer-based LiDAR object detection on the competitive nuScenes benchmark
and showcase the benefits of input-dependent multimodal query initialization,
while being more efficient than the available alternatives for LiDAR-camera
initialization. The proposed method can be applied with any combination of
sensor modalities as input, demonstrating its modularity.