Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection
Linyan Huang Zhiqi Li Chonghao Sima Wenhai Wang Jingdong Wang Yu Qiao Hongyang Li
Current research is primarily dedicated to advancing the accuracy of
camera-only 3D object detectors (apprentice) through the knowledge transferred
from LiDAR- or multi-modal-based counterparts (expert). However, the presence
of the domain gap between LiDAR and camera features, coupled with the inherent
incompatibility in temporal fusion, significantly hinders the effectiveness of
distillation-based enhancements for apprentices. Motivated by the success of
uni-modal distillation, an apprentice-friendly expert model would predominantly
rely on camera features, while still achieving comparable performance to
multi-modal models. To this end, we introduce VCD, a framework to improve the
camera-only apprentice model, including an apprentice-friendly multi-modal
expert and temporal-fusion-friendly distillation supervision. The multi-modal
expert VCD-E adopts an identical structure as that of the camera-only
apprentice in order to alleviate the feature disparity, and leverages LiDAR
input as a depth prior to reconstruct the 3D scene, achieving the performance
on par with other heterogeneous multi-modal experts. Additionally, a
fine-grained trajectory-based distillation module is introduced with the
purpose of individually rectifying the motion misalignment for each object in
the scene. With those improvements, our camera-only apprentice VCD-A sets new
state-of-the-art on nuScenes with a score of 63.1% NDS.