Exploring the Transfer Learning Capabilities of CLIP in Domain
  Generalization for Diabetic Retinopathy
Sanoojan Baliah Fadillah A. Maani Santosh Sanjeev Muhammad Haris Khan
Diabetic Retinopathy (DR), a leading cause of vision impairment, requires
early detection and treatment. Developing robust AI models for DR
classification holds substantial potential, but a key challenge is ensuring
their generalization in unfamiliar domains with varying data distributions. To
address this, our paper investigates cross-domain generalization, also known as
domain generalization (DG), within the context of DR classification. DG, a
challenging problem in the medical domain, is complicated by the difficulty of
gathering labeled data across different domains, such as patient demographics
and disease stages. Some recent studies have shown the effectiveness of using
CLIP to handle the DG problem in natural images. In this study, we investigate
CLIP's transfer learning capabilities and its potential for cross-domain
generalization in diabetic retinopathy (DR) classification. We carry out
comprehensive experiments to assess the efficacy and potential of CLIP in
addressing DG for DR classification. Further, we introduce a multi-modal
fine-tuning strategy named Context Optimization with Learnable Visual Tokens
(CoOpLVT), which enhances context optimization by conditioning on visual
features. Our findings demonstrate that the proposed method increases the
F1-score by 1.8% over the baseline, thus underlining its promise for effective
DG in DR classification. Our code is publicly available at
https://github.com/Sanoojan/CLIP-DRDG.