Make-A-Volume: Leveraging Latent Diffusion Models for Cross-Modality 3D
  Brain MRI Synthesis
Lingting Zhu Zeyue Xue Zhenchao Jin Xian Liu Jingzhen He Ziwei Liu Lequan Yu
Cross-modality medical image synthesis is a critical topic and has the
potential to facilitate numerous applications in the medical imaging field.
Despite recent successes in deep-learning-based generative models, most current
medical image synthesis methods rely on generative adversarial networks and
suffer from notorious mode collapse and unstable training. Moreover, the 2D
backbone-driven approaches would easily result in volumetric inconsistency,
while 3D backbones are challenging and impractical due to the tremendous memory
cost and training difficulty. In this paper, we introduce a new paradigm for
volumetric medical data synthesis by leveraging 2D backbones and present a
diffusion-based framework, Make-A-Volume, for cross-modality 3D medical image
synthesis. To learn the cross-modality slice-wise mapping, we employ a latent
diffusion model and learn a low-dimensional latent space, resulting in high
computational efficiency. To enable the 3D image synthesis and mitigate
volumetric inconsistency, we further insert a series of volumetric layers in
the 2D slice-mapping model and fine-tune them with paired 3D data. This
paradigm extends the 2D image diffusion model to a volumetric version with a
slightly increasing number of parameters and computation, offering a principled
solution for generic cross-modality 3D medical image synthesis. We showcase the
effectiveness of our Make-A-Volume framework on an in-house SWI-MRA brain MRI
dataset and a public T1-T2 brain MRI dataset. Experimental results demonstrate
that our framework achieves superior synthesis results with volumetric
consistency.