A Simple and Robust Framework for Cross-Modality Medical Image
  Segmentation applied to Vision Transformers
Matteo Bastico David Ryckelynck Laurent Corté Yannick Tillier Etienne Decencière
When it comes to clinical images, automatic segmentation has a wide variety
of applications and a considerable diversity of input domains, such as
different types of Magnetic Resonance Images (MRIs) and Computerized Tomography
(CT) scans. This heterogeneity is a challenge for cross-modality algorithms
that should equally perform independently of the input image type fed to them.
Often, segmentation models are trained using a single modality, preventing
generalization to other types of input data without resorting to transfer
learning techniques. Furthermore, the multi-modal or cross-modality
architectures proposed in the literature frequently require registered images,
which are not easy to collect in clinical environments, or need additional
processing steps, such as synthetic image generation. In this work, we propose
a simple framework to achieve fair image segmentation of multiple modalities
using a single conditional model that adapts its normalization layers based on
the input type, trained with non-registered interleaved mixed data. We show
that our framework outperforms other cross-modality segmentation methods, when
applied to the same 3D UNet baseline model, on the Multi-Modality Whole Heart
Segmentation Challenge. Furthermore, we define the Conditional Vision
Transformer (C-ViT) encoder, based on the proposed cross-modality framework,
and we show that it brings significant improvements to the resulting
segmentation, up to 6.87\% of Dice accuracy, with respect to its baseline
reference. The code to reproduce our experiments and the trained model weights
are available at https://github.com/matteo-bastico/MI-Seg.