Modality Influence in Multimodal Machine Learning
Abdelhamid Haouhat Slimane Bellaouar Attia Nehar Hadda Cherroun
Multimodal Machine Learning has emerged as a prominent research direction
across various applications such as Sentiment Analysis, Emotion Recognition,
Machine Translation, Hate Speech Recognition, and Movie Genre Classification.
This approach has shown promising results by utilizing modern deep learning
architectures. Despite the achievements made, challenges remain in data
representation, alignment techniques, reasoning, generation, and quantification
within multimodal learning. Additionally, assumptions about the dominant role
of textual modality in decision-making have been made. However, limited
investigations have been conducted on the influence of different modalities in
Multimodal Machine Learning systems. This paper aims to address this gap by
studying the impact of each modality on multimodal learning tasks. The research
focuses on verifying presumptions and gaining insights into the usage of
different modalities. The main contribution of this work is the proposal of a
methodology to determine the effect of each modality on several Multimodal
Machine Learning models and datasets from various tasks. Specifically, the
study examines Multimodal Sentiment Analysis, Multimodal Emotion Recognition,
Multimodal Hate Speech Recognition, and Multimodal Disease Detection. The study
objectives include training SOTA MultiModal Machine Learning models with masked
modalities to evaluate their impact on performance. Furthermore, the research
aims to identify the most influential modality or set of modalities for each
task and draw conclusions for diverse multimodal classification tasks. By
undertaking these investigations, this research contributes to a better
understanding of the role of individual modalities in multi-modal learning and
provides valuable insights for future advancements in this field.