IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object
  Detection
Junbo Yin Jianbing Shen Runnan Chen Wei Li Ruigang Yang Pascal Frossard Wenguan Wang
Bird's eye view (BEV) representation has emerged as a dominant solution for
describing 3D space in autonomous driving scenarios. However, objects in the
BEV representation typically exhibit small sizes, and the associated point
cloud context is inherently sparse, which leads to great challenges for
reliable 3D perception. In this paper, we propose IS-Fusion, an innovative
multimodal fusion framework that jointly captures the Instance- and Scene-level
contextual information. IS-Fusion essentially differs from existing approaches
that only focus on the BEV scene-level fusion by explicitly incorporating
instance-level multimodal information, thus facilitating the instance-centric
tasks like 3D object detection. It comprises a Hierarchical Scene Fusion (HSF)
module and an Instance-Guided Fusion (IGF) module. HSF applies Point-to-Grid
and Grid-to-Region transformers to capture the multimodal scene context at
different granularities. IGF mines instance candidates, explores their
relationships, and aggregates the local multimodal context for each instance.
These instances then serve as guidance to enhance the scene feature and yield
an instance-aware BEV representation. On the challenging nuScenes benchmark,
IS-Fusion outperforms all the published multimodal works to date. Code is
available at: https://github.com/yinjunbo/IS-Fusion.