Deep Metric Loss for Multimodal Learning
Sehwan Moon Hyunju Lee
Multimodal learning often outperforms its unimodal counterparts by exploiting
unimodal contributions and cross-modal interactions. However, focusing only on
integrating multimodal features into a unified comprehensive representation
overlooks the unimodal characteristics. In real data, the contributions of
modalities can vary from instance to instance, and they often reinforce or
conflict with each other. In this study, we introduce a novel \text{MultiModal}
loss paradigm for multimodal learning, which subgroups instances according to
their unimodal contributions. \text{MultiModal} loss can prevent inefficient
learning caused by overfitting and efficiently optimize multimodal models. On
synthetic data, \text{MultiModal} loss demonstrates improved classification
performance by subgrouping difficult instances within certain modalities. On
four real multimodal datasets, our loss is empirically shown to improve the
performance of recent models. Ablation studies verify the effectiveness of our
loss. Additionally, we show that our loss generates a reliable prediction score
for each modality, which is essential for subgrouping. Our \text{MultiModal}
loss is a novel loss function to subgroup instances according to the
contribution of modalities in multimodal learning and is applicable to a
variety of multimodal models with unimodal decisions. Our code is available at
https://github.com/SehwanMoon/MultiModalLoss.