A 3D generative model of pathological multi-modal MR images and
  segmentations
Virginia Fernandez Walter Hugo Lopez Pinaya Pedro Borges Mark S. Graham Tom Vercauteren M. Jorge Cardoso
Generative modelling and synthetic data can be a surrogate for real medical
imaging datasets, whose scarcity and difficulty to share can be a nuisance when
delivering accurate deep learning models for healthcare applications. In recent
years, there has been an increased interest in using these models for data
augmentation and synthetic data sharing, using architectures such as generative
adversarial networks (GANs) or diffusion models (DMs). Nonetheless, the
application of synthetic data to tasks such as 3D magnetic resonance imaging
(MRI) segmentation remains limited due to the lack of labels associated with
the generated images. Moreover, many of the proposed generative MRI models lack
the ability to generate arbitrary modalities due to the absence of explicit
contrast conditioning. These limitations prevent the user from adjusting the
contrast and content of the images and obtaining more generalisable data for
training task-specific models. In this work, we propose brainSPADE3D, a 3D
generative model for brain MRI and associated segmentations, where the user can
condition on specific pathological phenotypes and contrasts. The proposed joint
imaging-segmentation generative model is shown to generate high-fidelity
synthetic images and associated segmentations, with the ability to combine
pathologies. We demonstrate how the model can alleviate issues with
segmentation model performance when unexpected pathologies are present in the
data.