RAD-DINO: Exploring Scalable Medical Image Encoders Beyond Text
  Supervision
Fernando Pérez-García Harshita Sharma Sam Bond-Taylor Kenza Bouzid Valentina Salvatelli Maximilian Ilse Shruthi Bannur Daniel C. Castro Anton Schwaighofer Matthew P. Lungren Maria Wetscherek Noel Codella Stephanie L. Hyland Javier Alvarez-Valle Ozan Oktay
Language-supervised pre-training has proven to be a valuable method for
extracting semantically meaningful features from images, serving as a
foundational element in multimodal systems within the computer vision and
medical imaging domains. However, resulting features are limited by the
information contained within the text. This is particularly problematic in
medical imaging, where radiologists' written findings focus on specific
observations; a challenge compounded by the scarcity of paired imaging-text
data due to concerns over leakage of personal health information. In this work,
we fundamentally challenge the prevailing reliance on language supervision for
learning general purpose biomedical imaging encoders. We introduce RAD-DINO, a
biomedical image encoder pre-trained solely on unimodal biomedical imaging data
that obtains similar or greater performance than state-of-the-art biomedical
language supervised models on a diverse range of benchmarks. Specifically, the
quality of learned representations is evaluated on standard imaging tasks
(classification and semantic segmentation), and a vision-language alignment
task (text report generation from images). To further demonstrate the drawback
of language supervision, we show that features from RAD-DINO correlate with
other medical records (e.g., sex or age) better than language-supervised
models, which are generally not mentioned in radiology reports. Finally, we
conduct a series of ablations determining the factors in RAD-DINO's
performance; notably, we observe that RAD-DINO's downstream performance scales
well with the quantity and diversity of training data, demonstrating that
image-only supervision is a scalable approach for training a foundational
biomedical image encoder.