Hypercomplex Multimodal Emotion Recognition from EEG and Peripheral
  Physiological Signals
Eleonora Lopez Eleonora Chiarantano Eleonora Grassucci Danilo Comminiello
Multimodal emotion recognition from physiological signals is receiving an
increasing amount of attention due to the impossibility to control them at will
unlike behavioral reactions, thus providing more reliable information. Existing
deep learning-based methods still rely on extracted handcrafted features, not
taking full advantage of the learning ability of neural networks, and often
adopt a single-modality approach, while human emotions are inherently expressed
in a multimodal way. In this paper, we propose a hypercomplex multimodal
network equipped with a novel fusion module comprising parameterized
hypercomplex multiplications. Indeed, by operating in a hypercomplex domain the
operations follow algebraic rules which allow to model latent relations among
learned feature dimensions for a more effective fusion step. We perform
classification of valence and arousal from electroencephalogram (EEG) and
peripheral physiological signals, employing the publicly available database
MAHNOB-HCI surpassing a multimodal state-of-the-art network. The code of our
work is freely available at https://github.com/ispamm/MHyEEG.