IMITATE: Clinical Prior Guided Hierarchical Vision-Language Pre-training
Che Liu Sibo Cheng Miaojing Shi Anand Shah Wenjia Bai Rossella Arcucci
In the field of medical Vision-Language Pre-training (VLP), significant
efforts have been devoted to deriving text and image features from both
clinical reports and associated medical images. However, most existing methods
may have overlooked the opportunity in leveraging the inherent hierarchical
structure of clinical reports, which are generally split into `findings' for
descriptive content and `impressions' for conclusive observation. Instead of
utilizing this rich, structured format, current medical VLP approaches often
simplify the report into either a unified entity or fragmented tokens. In this
work, we propose a novel clinical prior guided VLP framework named IMITATE to
learn the structure information from medical reports with hierarchical
vision-language alignment. The framework derives multi-level visual features
from the chest X-ray (CXR) images and separately aligns these features with the
descriptive and the conclusive text encoded in the hierarchical medical report.
Furthermore, a new clinical-informed contrastive loss is introduced for
cross-modal learning, which accounts for clinical prior knowledge in
formulating sample correlations in contrastive learning. The proposed model,
IMITATE, outperforms baseline VLP methods across six different datasets,
spanning five medical imaging downstream tasks. Comprehensive experimental
results highlight the advantages of integrating the hierarchical structure of
medical reports for vision-language alignment.