SimPB: A Single Model for 2D and 3D Object Detection from Multiple
  Cameras
Yingqi Tang Zhaotie Meng Guoliang Chen Erkang Cheng
The field of autonomous driving has attracted considerable interest in
approaches that directly infer 3D objects in the Bird's Eye View (BEV) from
multiple cameras. Some attempts have also explored utilizing 2D detectors from
single images to enhance the performance of 3D detection. However, these
approaches rely on a two-stage process with separate detectors, where the 2D
detection results are utilized only once for token selection or query
initialization. In this paper, we present a single model termed SimPB, which
simultaneously detects 2D objects in the perspective view and 3D objects in the
BEV space from multiple cameras. To achieve this, we introduce a hybrid decoder
consisting of several multi-view 2D decoder layers and several 3D decoder
layers, specifically designed for their respective detection tasks. A Dynamic
Query Allocation module and an Adaptive Query Aggregation module are proposed
to continuously update and refine the interaction between 2D and 3D results, in
a cyclic 3D-2D-3D manner. Additionally, Query-group Attention is utilized to
strengthen the interaction among 2D queries within each camera group. In the
experiments, we evaluate our method on the nuScenes dataset and demonstrate
promising results for both 2D and 3D detection tasks. Our code is available at:
https://github.com/nullmax-vision/SimPB.