Gradient-Guided Modality Decoupling for Missing-Modality Robustness
Hao Wang Shengda Luo Guosheng Hu Jianguo Zhang
Multimodal learning with incomplete input data (missing modality) is
practical and challenging. In this work, we conduct an in-depth analysis of
this challenge and find that modality dominance has a significant negative
impact on the model training, greatly degrading the missing modality
performance. Motivated by Grad-CAM, we introduce a novel indicator, gradients,
to monitor and reduce modality dominance which widely exists in the
missing-modality scenario. In aid of this indicator, we present a novel
Gradient-guided Modality Decoupling (GMD) method to decouple the dependency on
dominating modalities. Specifically, GMD removes the conflicted gradient
components from different modalities to achieve this decoupling, significantly
improving the performance. In addition, to flexibly handle modal-incomplete
data, we design a parameter-efficient Dynamic Sharing (DS) framework which can
adaptively switch on/off the network parameters based on whether one modality
is available. We conduct extensive experiments on three popular multimodal
benchmarks, including BraTS 2018 for medical segmentation, CMU-MOSI, and
CMU-MOSEI for sentiment analysis. The results show that our method can
significantly outperform the competitors, showing the effectiveness of the
proposed solutions. Our code is released here:
https://github.com/HaoWang420/Gradient-guided-Modality-Decoupling.