GuideGen: A Text-guided Framework for Joint CT Volume and Anatomical
  structure Generation
Linrui Dai Rongzhao Zhang Zhongzhen Huang Xiaofan Zhang
The annotation burden and extensive labor for gathering a large medical
dataset with images and corresponding labels are rarely cost-effective and
highly intimidating. This results in a lack of abundant training data that
undermines downstream tasks and partially contributes to the challenge image
analysis faces in the medical field. As a workaround, given the recent success
of generative neural models, it is now possible to synthesize image datasets at
a high fidelity guided by external constraints. This paper explores this
possibility and presents \textbf{GuideGen}: a pipeline that jointly generates
CT images and tissue masks for abdominal organs and colorectal cancer
conditioned on a text prompt. Firstly, we introduce Volumetric Mask Sampler to
fit the discrete distribution of mask labels and generate low-resolution 3D
tissue masks. Secondly, our Conditional Image Generator autoregressively
generates CT slices conditioned on a corresponding mask slice to incorporate
both style information and anatomical guidance. This pipeline guarantees high
fidelity and variability as well as exact alignment between generated CT
volumes and tissue masks. Both qualitative and quantitative experiments on 3D
abdominal CTs demonstrate a high performance of our proposed pipeline, thereby
proving our method can serve as a dataset generator and provide potential
benefits to downstream tasks. It is hoped that our work will offer a promising
solution on the multimodality generation of CT and its anatomical mask. Our
source code is publicly available at
https://github.com/OvO1111/JointImageGeneration.