A Multimodal Intermediate Fusion Network with Manifold Learning for
  Stress Detection
Morteza Bodaghi Majid Hosseini Raju Gottumukkala
Multimodal deep learning methods capture synergistic features from multiple
modalities and have the potential to improve accuracy for stress detection
compared to unimodal methods. However, this accuracy gain typically comes from
high computational cost due to the high-dimensional feature spaces, especially
for intermediate fusion. Dimensionality reduction is one way to optimize
multimodal learning by simplifying data and making the features more amenable
to processing and analysis, thereby reducing computational complexity. This
paper introduces an intermediate multimodal fusion network with manifold
learning-based dimensionality reduction. The multimodal network generates
independent representations from biometric signals and facial landmarks through
1D-CNN and 2D-CNN. Finally, these features are fused and fed to another 1D-CNN
layer, followed by a fully connected dense layer. We compared various
dimensionality reduction techniques for different variations of unimodal and
multimodal networks. We observe that the intermediate-level fusion with the
Multi-Dimensional Scaling (MDS) manifold method showed promising results with
an accuracy of 96.00\% in a Leave-One-Subject-Out Cross-Validation (LOSO-CV)
paradigm over other dimensional reduction methods. MDS had the highest
computational cost among manifold learning methods. However, while
outperforming other networks, it managed to reduce the computational cost of
the proposed networks by 25\% when compared to six well-known conventional
feature selection methods used in the preprocessing step.