Anatomical Structure-Guided Medical Vision-Language Pre-training
Qingqiu Li Xiaohan Yan Jilan Xu Runtian Yuan Yuejie Zhang Rui Feng Quanli Shen Xiaobo Zhang Shujun Wang
Learning medical visual representations through vision-language pre-training
has reached remarkable progress. Despite the promising performance, it still
faces challenges, i.e., local alignment lacks interpretability and clinical
relevance, and the insufficient internal and external representation learning
of image-report pairs. To address these issues, we propose an Anatomical
Structure-Guided (ASG) framework. Specifically, we parse raw reports into
triplets <anatomical region, finding, existence>, and fully utilize each
element as supervision to enhance representation learning. For anatomical
region, we design an automatic anatomical region-sentence alignment paradigm in
collaboration with radiologists, considering them as the minimum semantic units
to explore fine-grained local alignment. For finding and existence, we regard
them as image tags, applying an image-tag recognition decoder to associate
image features with their respective tags within each sample and constructing
soft labels for contrastive learning to improve the semantic association of
different image-report pairs. We evaluate the proposed ASG framework on two
downstream tasks, including five public benchmarks. Experimental results
demonstrate that our method outperforms the state-of-the-art methods.