Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal
  Sentiment Space
Zhuyang Xie Yan Yang Jie Wang Xiaorong Liu Xiaofan Li
Multimodal video sentiment analysis aims to integrate multiple modal
information to analyze the opinions and attitudes of speakers. Most previous
work focuses on exploring the semantic interactions of intra- and
inter-modality. However, these works ignore the reliability of multimodality,
i.e., modalities tend to contain noise, semantic ambiguity, missing modalities,
etc. In addition, previous multimodal approaches treat different modalities
equally, largely ignoring their different contributions. Furthermore, existing
multimodal sentiment analysis methods directly regress sentiment scores without
considering ordinal relationships within sentiment categories, with limited
performance. To address the aforementioned problems, we propose a trustworthy
multimodal sentiment ordinal network (TMSON) to improve performance in
sentiment analysis. Specifically, we first devise a unimodal feature extractor
for each modality to obtain modality-specific features. Then, an uncertainty
distribution estimation network is customized, which estimates the unimodal
uncertainty distributions. Next, Bayesian fusion is performed on the learned
unimodal distributions to obtain multimodal distributions for sentiment
prediction. Finally, an ordinal-aware sentiment space is constructed, where
ordinal regression is used to constrain the multimodal distributions. Our
proposed TMSON outperforms baselines on multimodal sentiment analysis tasks,
and empirical results demonstrate that TMSON is capable of reducing uncertainty
to obtain more robust predictions.