Unified Medical Image-Text-Label Contrastive Learning With Continuous
  Prompt
Yuhao Wang
Contrastive language-image Pre-training (CLIP) [13] can leverage large
datasets of unlabeled Image-Text pairs, which have demonstrated impressive
performance in various downstream tasks. Given that annotating medical data is
time-consuming and laborious, Image-Text Pre-training has promising
applications in exploiting large-scale medical image and radiology report
datasets. However, medical Image-Text Pre-training faces several challenges, as
follows: (1) Due to privacy concerns, the amount of available medical data is
relatively small compared to natural data, leading to weaker generalization
ability of the model. (2) Medical images are highly similar with only
fine-grained differences in subtleties, resulting in a large number of
false-negative sample pairs in comparison learning. (3) The hand-crafted Prompt
usually differs from the natural medical image report, Subtle changes in
wording can lead to significant differences in performance. In this paper, we
propose a unified Image-Text-Label contrastive learning framework based on
continuous prompts, with three main contributions. First, We unified the data
of images, text, and labels, which greatly expanded the training data that the
model could utilize. Second, we address the issue of data diversity and the
impact of hand-crafted prompts on model performance by introducing continuous
implicit prompts. Lastly, we propose a ImageText-Label contrastive Training to
mitigate the problem of too many false-negative samples. We demonstrate through
sufficient experiments that the Unified Medical Contrastive Learning (UMCL)
framework exhibits excellent performance on several downstream tasks.