MeDSLIP: Medical Dual-Stream Language-Image Pre-training for
  Fine-grained Alignment
Wenrui Fan Mohammod Naimul Islam Suvon Shuo Zhou Xianyuan Liu Samer Alabed Venet Osmani Andrew Swift Chen Chen Haiping Lu
Vision-language pre-training (VLP) models have shown significant advancements
in the medical domain. Yet, most VLP models align raw reports to images at a
very coarse level, without modeling fine-grained relationships between
anatomical and pathological concepts outlined in reports and the corresponding
semantic counterparts in images. To address this problem, we propose a Medical
Dual-Stream Language-Image Pre-training (MeDSLIP) framework. Specifically,
MeDSLIP establishes vision-language fine-grained alignments via disentangling
visual and textual representations into anatomy-relevant and pathology-relevant
streams. Moreover, a novel vision-language Prototypical Contr-astive Learning
(ProtoCL) method is adopted in MeDSLIP to enhance the alignment within the
anatomical and pathological streams. MeDSLIP further employs cross-stream
Intra-image Contrastive Learning (ICL) to ensure the consistent coexistence of
paired anatomical and pathological concepts within the same image. Such a
cross-stream regularization encourages the model to exploit the synchrony
between two streams for a more comprehensive representation learning. MeDSLIP
is evaluated under zero-shot and supervised fine-tuning settings on three
public datasets: NIH CXR14, RSNA Pneumonia, and SIIM-ACR Pneumothorax. Under
these settings, MeDSLIP outperforms six leading CNN-based models on
classification, grounding, and segmentation tasks.