Boosting Dermatoscopic Lesion Segmentation via Diffusion Models with
  Visual and Textual Prompts
Shiyi Du Xiaosong Wang Yongyi Lu Yuyin Zhou Shaoting Zhang Alan Yuille Kang Li Zongwei Zhou
Image synthesis approaches, e.g., generative adversarial networks, have been
popular as a form of data augmentation in medical image analysis tasks. It is
primarily beneficial to overcome the shortage of publicly accessible data and
associated quality annotations. However, the current techniques often lack
control over the detailed contents in generated images, e.g., the type of
disease patterns, the location of lesions, and attributes of the diagnosis. In
this work, we adapt the latest advance in the generative model, i.e., the
diffusion model, with the added control flow using lesion-specific visual and
textual prompts for generating dermatoscopic images. We further demonstrate the
advantage of our diffusion model-based framework over the classical generation
models in both the image quality and boosting the segmentation performance on
skin lesions. It can achieve a 9% increase in the SSIM image quality measure
and an over 5% increase in Dice coefficients over the prior arts.