Modality-Collaborative Transformer with Hybrid Feature Reconstruction
  for Robust Emotion Recognition
Chengxin Chen Pengyuan Zhang
As a vital aspect of affective computing, Multimodal Emotion Recognition has
been an active research area in the multimedia community. Despite recent
progress, this field still confronts two major challenges in real-world
applications: 1) improving the efficiency of constructing joint representations
from unaligned multimodal features, and 2) relieving the performance decline
caused by random modality feature missing. In this paper, we propose a unified
framework, Modality-Collaborative Transformer with Hybrid Feature
Reconstruction (MCT-HFR), to address these issues. The crucial component of MCT
is a novel attention-based encoder which concurrently extracts and dynamically
balances the intra- and inter-modality relations for all associated modalities.
With additional modality-wise parameter sharing, a more compact representation
can be encoded with less time and space complexity. To improve the robustness
of MCT, we further introduce HFR which consists of two modules: Local Feature
Imagination (LFI) and Global Feature Alignment (GFA). During model training,
LFI leverages complete features as supervisory signals to recover local missing
features, while GFA is designed to reduce the global semantic gap between
pairwise complete and incomplete representations. Experimental evaluations on
two popular benchmark datasets demonstrate that our proposed method
consistently outperforms advanced baselines in both complete and incomplete
data scenarios.