Exploring the Transferability of a Foundation Model for Fundus Images:
  Application to Hypertensive Retinopathy
Julio Silva-Rodriguez Jihed Chelbi Waziha Kabir Hadi Chakor Jose Dolz Ismail Ben Ayed Riadh Kobbi
Using deep learning models pre-trained on Imagenet is the traditional
solution for medical image classification to deal with data scarcity.
Nevertheless, relevant literature supports that this strategy may offer limited
gains due to the high dissimilarity between domains. Currently, the paradigm of
adapting domain-specialized foundation models is proving to be a promising
alternative. However, how to perform such knowledge transfer, and the benefits
and limitations it presents, are under study. The CGI-HRDC challenge for
Hypertensive Retinopathy diagnosis on fundus images introduces an appealing
opportunity to evaluate the transferability of a recently released
vision-language foundation model of the retina, FLAIR. In this work, we explore
the potential of using FLAIR features as starting point for fundus image
classification, and we compare its performance with regard to Imagenet
initialization on two popular transfer learning methods: Linear Probing (LP)
and Fine-Tuning (FP). Our empirical observations suggest that, in any case, the
use of the traditional strategy provides performance gains. In contrast, direct
transferability from FLAIR model allows gains of 2.5%. When fine-tuning the
whole network, the performance gap increases up to 4%. In this case, we show
that avoiding feature deterioration via LP initialization of the classifier
allows the best re-use of the rich pre-trained features. Although direct
transferability using LP still offers limited performance, we believe that
foundation models such as FLAIR will drive the evolution of deep-learning-based
fundus image analysis.