Multi-Dimension-Embedding-Aware Modality Fusion Transformer for
  Psychiatric Disorder Clasification
Guoxin Wang Xuyang Cao Shan An Fengmei Fan Chao Zhang Jinsong Wang Feng Yu Zhiren Wang
Deep learning approaches, together with neuroimaging techniques, play an
important role in psychiatric disorders classification. Previous studies on
psychiatric disorders diagnosis mainly focus on using functional connectivity
matrices of resting-state functional magnetic resonance imaging (rs-fMRI) as
input, which still needs to fully utilize the rich temporal information of the
time series of rs-fMRI data. In this work, we proposed a
multi-dimension-embedding-aware modality fusion transformer (MFFormer) for
schizophrenia and bipolar disorder classification using rs-fMRI and T1 weighted
structural MRI (T1w sMRI). Concretely, to fully utilize the temporal
information of rs-fMRI and spatial information of sMRI, we constructed a deep
learning architecture that takes as input 2D time series of rs-fMRI and 3D
volumes T1w. Furthermore, to promote intra-modality attention and information
fusion across different modalities, a fusion transformer module (FTM) is
designed through extensive self-attention of hybrid feature maps of
multi-modality. In addition, a dimension-up and dimension-down strategy is
suggested to properly align feature maps of multi-dimensional from different
modalities. Experimental results on our private and public OpenfMRI datasets
show that our proposed MFFormer performs better than that using a single
modality or multi-modality MRI on schizophrenia and bipolar disorder diagnosis.