One-stage Modality Distillation for Incomplete Multimodal Learning
Shicai Wei Yang Luo Chunbo Luo
Learning based on multimodal data has attracted increasing interest recently.
While a variety of sensory modalities can be collected for training, not all of
them are always available in development scenarios, which raises the challenge
to infer with incomplete modality. To address this issue, this paper presents a
one-stage modality distillation framework that unifies the privileged knowledge
transfer and modality information fusion into a single optimization procedure
via multi-task learning. Compared with the conventional modality distillation
that performs them independently, this helps to capture the valuable
representation that can assist the final model inference directly.
Specifically, we propose the joint adaptation network for the modality transfer
task to preserve the privileged information. This addresses the representation
heterogeneity caused by input discrepancy via the joint distribution
adaptation. Then, we introduce the cross translation network for the modality
fusion task to aggregate the restored and available modality features. It
leverages the parameters-sharing strategy to capture the cross-modal cues
explicitly. Extensive experiments on RGB-D classification and segmentation
tasks demonstrate the proposed multimodal inheritance framework can overcome
the problem of incomplete modality input in various scenes and achieve
state-of-the-art performance.