Enhancing the vision-language foundation model with key semantic
  knowledge-emphasized report refinement
Cheng Li Weijian Huang Hao Yang Jiarun Liu Shanshan Wang
Recently, vision-language representation learning has made remarkable
advancements in building up medical foundation models, holding immense
potential for transforming the landscape of clinical research and medical care.
The underlying hypothesis is that the rich knowledge embedded in radiology
reports can effectively assist and guide the learning process, reducing the
need for additional labels. However, these reports tend to be complex and
sometimes even consist of redundant descriptions that make the representation
learning too challenging to capture the key semantic information. This paper
develops a novel iterative vision-language representation learning framework by
proposing a key semantic knowledge-emphasized report refinement method.
Particularly, raw radiology reports are refined to highlight the key
information according to a constructed clinical dictionary and two
model-optimized knowledge-enhancement metrics. The iterative framework is
designed to progressively learn, starting from gaining a general understanding
of the patient's condition based on raw reports and gradually refines and
extracts critical information essential to the fine-grained analysis tasks. The
effectiveness of the proposed framework is validated on various downstream
medical image analysis tasks, including disease classification,
region-of-interest segmentation, and phrase grounding. Our framework surpasses
seven state-of-the-art methods in both fine-tuning and zero-shot settings,
demonstrating its encouraging potential for different clinical applications.