Self-MI: Efficient Multimodal Fusion via Self-Supervised Multi-Task
  Learning with Auxiliary Mutual Information Maximization
Cam-Van Thi Nguyen Ngoc-Hoa Thi Nguyen Duc-Trong Le Quang-Thuy Ha
Multimodal representation learning poses significant challenges in capturing
informative and distinct features from multiple modalities. Existing methods
often struggle to exploit the unique characteristics of each modality due to
unified multimodal annotations. In this study, we propose Self-MI in the
self-supervised learning fashion, which also leverage Contrastive Predictive
Coding (CPC) as an auxiliary technique to maximize the Mutual Information (MI)
between unimodal input pairs and the multimodal fusion result with unimodal
inputs. Moreover, we design a label generation module, $ULG_{MI}$ for short,
that enables us to create meaningful and informative labels for each modality
in a self-supervised manner. By maximizing the Mutual Information, we encourage
better alignment between the multimodal fusion and the individual modalities,
facilitating improved multimodal fusion. Extensive experiments on three
benchmark datasets including CMU-MOSI, CMU-MOSEI, and SIMS, demonstrate the
effectiveness of Self-MI in enhancing the multimodal fusion task.