High-Quality Medical Image Generation from Free-hand Sketch
Quan Huu Cap Atsushi Fukuda
Generating medical images from human-drawn free-hand sketches holds promise
for various important medical imaging applications. Due to the extreme
difficulty in collecting free-hand sketch data in the medical domain, most deep
learning-based methods have been proposed to generate medical images from the
synthesized sketches (e.g., edge maps or contours of segmentation masks from
real images). However, these models often fail to generalize on the free-hand
sketches, leading to unsatisfactory results. In this paper, we propose a
practical free-hand sketch-to-image generation model called Sketch2MedI that
learns to represent sketches in StyleGAN's latent space and generate medical
images from it. Thanks to the ability to encode sketches into this meaningful
representation space, Sketch2MedI only requires synthesized sketches for
training, enabling a cost-effective learning process. Our Sketch2MedI
demonstrates a robust generalization to free-hand sketches, resulting in
high-quality and realistic medical image generations. Comparative evaluations
of Sketch2MedI against the pix2pix, CycleGAN, UNIT, and U-GAT-IT models show
superior performance in generating pharyngeal images, both quantitative and
qualitative across various metrics.