MMOE: Mixture of Multimodal Interaction Experts
Haofei Yu Paul Pu Liang Ruslan Salakhutdinov Louis-Philippe Morency
Multimodal machine learning, which studies the information and interactions
across various input modalities, has made significant advancements in
understanding the relationship between images and descriptive text. However,
this is just a portion of the potential multimodal interactions seen in the
real world and does not include new interactions between conflicting utterances
and gestures in predicting sarcasm, for example. Notably, the current methods
for capturing shared information often do not extend well to these more nuanced
interactions, sometimes performing as low as 50% in binary classification. In
this paper, we address this problem via a new approach called MMOE, which
stands for a mixture of multimodal interaction experts. Our method
automatically classifies data points from unlabeled multimodal datasets by
their interaction type and employs specialized models for each specific
interaction. Based on our experiments, this approach improves performance on
these challenging interactions by more than 10%, leading to an overall increase
of 2% for tasks like sarcasm prediction. As a result, interaction
quantification provides new insights for dataset analysis and yields simple
approaches that obtain state-of-the-art performance.