Representation Disparity-aware Distillation for 3D Object Detection
Yanjing Li Sheng Xu Mingbao Lin Jihao Yin Baochang Zhang Xianbin Cao
In this paper, we focus on developing knowledge distillation (KD) for compact
3D detectors. We observe that off-the-shelf KD methods manifest their efficacy
only when the teacher model and student counterpart share similar intermediate
feature representations. This might explain why they are less effective in
building extreme-compact 3D detectors where significant representation
disparity arises due primarily to the intrinsic sparsity and irregularity in 3D
point clouds. This paper presents a novel representation disparity-aware
distillation (RDD) method to address the representation disparity issue and
reduce performance gap between compact students and over-parameterized
teachers. This is accomplished by building our RDD from an innovative
perspective of information bottleneck (IB), which can effectively minimize the
disparity of proposal region pairs from student and teacher in features and
logits. Extensive experiments are performed to demonstrate the superiority of
our RDD over existing KD methods. For example, our RDD increases mAP of
CP-Voxel-S to 57.1% on nuScenes dataset, which even surpasses teacher
performance while taking up only 42% FLOPs.