Enhancing medical vision-language contrastive learning via
  inter-matching relation modelling
Mingjian Li Mingyuan Meng Michael Fulham David Dagan Feng Lei Bi Jinman Kim
Medical image representations can be learned through medical vision-language
contrastive learning (mVLCL) where medical imaging reports are used as weak
supervision through image-text alignment. These learned image representations
can be transferred to and benefit various downstream medical vision tasks such
as disease classification and segmentation. Recent mVLCL methods attempt to
align image sub-regions and the report keywords as local-matchings. However,
these methods aggregate all local-matchings via simple pooling operations while
ignoring the inherent relations between them. These methods therefore fail to
reason between local-matchings that are semantically related, e.g.,
local-matchings that correspond to the disease word and the location word
(semantic-relations), and also fail to differentiate such clinically important
local-matchings from others that correspond to less meaningful words, e.g.,
conjunction words (importance-relations). Hence, we propose a mVLCL method that
models the inter-matching relations between local-matchings via a
relation-enhanced contrastive learning framework (RECLF). In RECLF, we
introduce a semantic-relation reasoning module (SRM) and an importance-relation
reasoning module (IRM) to enable more fine-grained report supervision for image
representation learning. We evaluated our method using four public benchmark
datasets on four downstream tasks, including segmentation, zero-shot
classification, supervised classification, and cross-modal retrieval. Our
results demonstrated the superiority of our RECLF over the state-of-the-art
mVLCL methods with consistent improvements across single-modal and cross-modal
tasks. These results suggest that our RECLF, by modelling the inter-matching
relations, can learn improved medical image representations with better
generalization capabilities.