MAC: ModAlity Calibration for Object Detection
Yutian Lei Jun Liu Dong Huang
The flourishing success of Deep Neural Networks(DNNs) on RGB-input perception
tasks has opened unbounded possibilities for non-RGB-input perception tasks,
such as object detection from wireless signals, lidar scans, and infrared
images. Compared to the matured development pipeline of RGB-input (source
modality) models, developing non-RGB-input (target-modality) models from
scratch poses excessive challenges in the modality-specific network
design/training tricks and labor in the target-modality annotation. In this
paper, we propose ModAlity Calibration (MAC), an efficient pipeline for
calibrating target-modality inputs to the DNN object detection models developed
on the RGB (source) modality. We compose a target-modality-input model by
adding a small calibrator module ahead of a source-modality model and introduce
MAC training techniques to impose dense supervision on the calibrator. By
leveraging (1) prior knowledge synthesized from the source-modality model and
(2) paired {target, source} data with zero manual annotations, our
target-modality models reach comparable or better metrics than baseline models
that require 100% manual annotations. We demonstrate the effectiveness of MAC
by composing the WiFi-input, Lidar-input, and Thermal-Infrared-input models
upon the pre-trained RGB-input models respectively.