Joint Self-Supervised and Supervised Contrastive Learning for Multimodal
  MRI Data: Towards Predicting Abnormal Neurodevelopment
Zhiyuan Li Hailong Li Anca L. Ralescu Jonathan R. Dillman Mekibib Altaye Kim M. Cecil Nehal A. Parikh Lili He
The integration of different imaging modalities, such as structural,
diffusion tensor, and functional magnetic resonance imaging, with deep learning
models has yielded promising outcomes in discerning phenotypic characteristics
and enhancing disease diagnosis. The development of such a technique hinges on
the efficient fusion of heterogeneous multimodal features, which initially
reside within distinct representation spaces. Naively fusing the multimodal
features does not adequately capture the complementary information and could
even produce redundancy. In this work, we present a novel joint self-supervised
and supervised contrastive learning method to learn the robust latent feature
representation from multimodal MRI data, allowing the projection of
heterogeneous features into a shared common space, and thereby amalgamating
both complementary and analogous information across various modalities and
among similar subjects. We performed a comparative analysis between our
proposed method and alternative deep multimodal learning approaches. Through
extensive experiments on two independent datasets, the results demonstrated
that our method is significantly superior to several other deep multimodal
learning methods in predicting abnormal neurodevelopment. Our method has the
capability to facilitate computer-aided diagnosis within clinical practice,
harnessing the power of multimodal data.