Why does my medical AI look at pictures of birds? Exploring the efficacy
  of transfer learning across domain boundaries
Frederic Jonske Moon Kim Enrico Nasca Janis Evers Johannes Haubold René Hosch Felix Nensa Michael Kamp Constantin Seibold Jan Egger Jens Kleesiek
It is an open secret that ImageNet is treated as the panacea of pretraining.
Particularly in medical machine learning, models not trained from scratch are
often finetuned based on ImageNet-pretrained models. We posit that pretraining
on data from the domain of the downstream task should almost always be
preferred instead. We leverage RadNet-12M, a dataset containing more than 12
million computed tomography (CT) image slices, to explore the efficacy of
self-supervised pretraining on medical and natural images. Our experiments
cover intra- and cross-domain transfer scenarios, varying data scales,
finetuning vs. linear evaluation, and feature space analysis. We observe that
intra-domain transfer compares favorably to cross-domain transfer, achieving
comparable or improved performance (0.44% - 2.07% performance increase using
RadNet pretraining, depending on the experiment) and demonstrate the existence
of a domain boundary-related generalization gap and domain-specific learned
features.