FULLER: Unified Multi-modality Multi-task 3D Perception via Multi-level
  Gradient Calibration
Zhijian Huang Sihao Lin Guiyu Liu Mukun Luo Chaoqiang Ye Hang Xu Xiaojun Chang Xiaodan Liang
Multi-modality fusion and multi-task learning are becoming trendy in 3D
autonomous driving scenario, considering robust prediction and computation
budget. However, naively extending the existing framework to the domain of
multi-modality multi-task learning remains ineffective and even poisonous due
to the notorious modality bias and task conflict. Previous works manually
coordinate the learning framework with empirical knowledge, which may lead to
sub-optima. To mitigate the issue, we propose a novel yet simple multi-level
gradient calibration learning framework across tasks and modalities during
optimization. Specifically, the gradients, produced by the task heads and used
to update the shared backbone, will be calibrated at the backbone's last layer
to alleviate the task conflict. Before the calibrated gradients are further
propagated to the modality branches of the backbone, their magnitudes will be
calibrated again to the same level, ensuring the downstream tasks pay balanced
attention to different modalities. Experiments on large-scale benchmark
nuScenes demonstrate the effectiveness of the proposed method, eg, an absolute
14.4% mIoU improvement on map segmentation and 1.4% mAP improvement on 3D
detection, advancing the application of 3D autonomous driving in the domain of
multi-modality fusion and multi-task learning. We also discuss the links
between modalities and tasks.