Disentangled Multimodal Brain MR Image Translation via Transformer-based
  Modality Infuser
Jihoon Cho Xiaofeng Liu Fangxu Xing Jinsong Ouyang Georges El Fakhri Jinah Park Jonghye Woo
Multimodal Magnetic Resonance (MR) Imaging plays a crucial role in disease
diagnosis due to its ability to provide complementary information by analyzing
a relationship between multimodal images on the same subject. Acquiring all MR
modalities, however, can be expensive, and, during a scanning session, certain
MR images may be missed depending on the study protocol. The typical solution
would be to synthesize the missing modalities from the acquired images such as
using generative adversarial networks (GANs). Yet, GANs constructed with
convolutional neural networks (CNNs) are likely to suffer from a lack of global
relationships and mechanisms to condition the desired modality. To address
this, in this work, we propose a transformer-based modality infuser designed to
synthesize multimodal brain MR images. In our method, we extract
modality-agnostic features from the encoder and then transform them into
modality-specific features using the modality infuser. Furthermore, the
modality infuser captures long-range relationships among all brain structures,
leading to the generation of more realistic images. We carried out experiments
on the BraTS 2018 dataset, translating between four MR modalities, and our
experimental results demonstrate the superiority of our proposed method in
terms of synthesis quality. In addition, we conducted experiments on a brain
tumor segmentation task and different conditioning methods.