Learning Better Contrastive View from Radiologist's Gaze
Sheng Wang Zixu Zhuang Xi Ouyang Lichi Zhang Zheren Li Chong Ma Tianming Liu Dinggang Shen Qian Wang
Recent self-supervised contrastive learning methods greatly benefit from the
Siamese structure that aims to minimizing distances between positive pairs.
These methods usually apply random data augmentation to input images, expecting
the augmented views of the same images to be similar and positively paired.
However, random augmentation may overlook image semantic information and
degrade the quality of augmented views in contrastive learning. This issue
becomes more challenging in medical images since the abnormalities related to
diseases can be tiny, and are easy to be corrupted (e.g., being cropped out) in
the current scheme of random augmentation. In this work, we first demonstrate
that, for widely-used X-ray images, the conventional augmentation prevalent in
contrastive pre-training can affect the performance of the downstream diagnosis
or classification tasks. Then, we propose a novel augmentation method, i.e.,
FocusContrast, to learn from radiologists' gaze in diagnosis and generate
contrastive views for medical images with guidance from radiologists' visual
attention. Specifically, we track the gaze movement of radiologists and model
their visual attention when reading to diagnose X-ray images. The learned model
can predict visual attention of the radiologists given a new input image, and
further guide the attention-aware augmentation that hardly neglects the
disease-related abnormalities. As a plug-and-play and framework-agnostic
module, FocusContrast consistently improves state-of-the-art contrastive
learning methods of SimCLR, MoCo, and BYOL by 4.0~7.0% in classification
accuracy on a knee X-ray dataset.