Neuro-Inspired Information-Theoretic Hierarchical Perception for
  Multimodal Learning
Xiongye Xiao Gengshuo Liu Gaurav Gupta Defu Cao Shixuan Li Yaxing Li Tianqing Fang Mingxi Cheng Paul Bogdan
Integrating and processing information from various sources or modalities are
critical for obtaining a comprehensive and accurate perception of the real
world in autonomous systems and cyber-physical systems. Drawing inspiration
from neuroscience, we develop the Information-Theoretic Hierarchical Perception
(ITHP) model, which utilizes the concept of information bottleneck. Different
from most traditional fusion models that incorporate all modalities identically
in neural networks, our model designates a prime modality and regards the
remaining modalities as detectors in the information pathway, serving to
distill the flow of information. Our proposed perception model focuses on
constructing an effective and compact information flow by achieving a balance
between the minimization of mutual information between the latent state and the
input modal state, and the maximization of mutual information between the
latent states and the remaining modal states. This approach leads to compact
latent state representations that retain relevant information while minimizing
redundancy, thereby substantially enhancing the performance of multimodal
representation learning. Experimental evaluations on the MUStARD, CMU-MOSI, and
CMU-MOSEI datasets demonstrate that our model consistently distills crucial
information in multimodal learning scenarios, outperforming state-of-the-art
benchmarks. Remarkably, on the CMU-MOSI dataset, ITHP surpasses human-level
performance in the multimodal sentiment binary classification task across all
evaluation metrics (i.e., Binary Accuracy, F1 Score, Mean Absolute Error, and
Pearson Correlation).