Revolutionizing Disease Diagnosis with simultaneous functional PET/MR
  and Deeply Integrated Brain Metabolic, Hemodynamic, and Perfusion Networks
Luoyu Wang Yitian Tao Qing Yang Yan Liang Siwei Liu Hongcheng Shi Dinggang Shen Han Zhang
Simultaneous functional PET/MR (sf-PET/MR) presents a cutting-edge multimodal
neuroimaging technique. It provides an unprecedented opportunity for
concurrently monitoring and integrating multifaceted brain networks built by
spatiotemporally covaried metabolic activity, neural activity, and cerebral
blood flow (perfusion). Albeit high scientific/clinical values, short in
hardware accessibility of PET/MR hinders its applications, let alone modern
AI-based PET/MR fusion models. Our objective is to develop a clinically
feasible AI-based disease diagnosis model trained on comprehensive sf-PET/MR
data with the power of, during inferencing, allowing single modality input
(e.g., PET only) as well as enforcing multimodal-based accuracy. To this end,
we propose MX-ARM, a multimodal MiXture-of-experts Alignment and Reconstruction
Model. It is modality detachable and exchangeable, allocating different
multi-layer perceptrons dynamically ("mixture of experts") through learnable
weights to learn respective representations from different modalities. Such
design will not sacrifice model performance in uni-modal situation. To fully
exploit the inherent complex and nonlinear relation among modalities while
producing fine-grained representations for uni-modal inference, we subsequently
add a modal alignment module to line up a dominant modality (e.g., PET) with
representations of auxiliary modalities (MR). We further adopt multimodal
reconstruction to promote the quality of learned features. Experiments on
precious multimodal sf-PET/MR data for Mild Cognitive Impairment diagnosis
showcase the efficacy of our model toward clinically feasible precision
medicine.