Fast and Accurate Transferability Measurement by Evaluating Intra-class
  Feature Variance
Huiwen Xu U Kang
Given a set of pre-trained models, how can we quickly and accurately find the
most useful pre-trained model for a downstream task? Transferability
measurement is to quantify how transferable is a pre-trained model learned on a
source task to a target task. It is used for quickly ranking pre-trained models
for a given task and thus becomes a crucial step for transfer learning.
Existing methods measure transferability as the discrimination ability of a
source model for a target data before transfer learning, which cannot
accurately estimate the fine-tuning performance. Some of them restrict the
application of transferability measurement in selecting the best supervised
pre-trained models that have classifiers. It is important to have a general
method for measuring transferability that can be applied in a variety of
situations, such as selecting the best self-supervised pre-trained models that
do not have classifiers, and selecting the best transferring layer for a target
task. In this work, we propose TMI (TRANSFERABILITY MEASUREMENT WITH
INTRA-CLASS FEATURE VARIANCE), a fast and accurate algorithm to measure
transferability. We view transferability as the generalization of a pre-trained
model on a target task by measuring intra-class feature variance. Intra-class
variance evaluates the adaptability of the model to a new task, which measures
how transferable the model is. Compared to previous studies that estimate how
discriminative the models are, intra-class variance is more accurate than those
as it does not require an optimal feature extractor and classifier. Extensive
experiments on real-world datasets show that TMI outperforms competitors for
selecting the top-5 best models, and exhibits consistently better correlation
in 13 out of 17 cases.