Unleashing the power of Neural Collapse for Transferability Estimation
Yuhe Ding Bo Jiang Lijun Sheng Aihua Zheng Jian Liang
Transferability estimation aims to provide heuristics for quantifying how
suitable a pre-trained model is for a specific downstream task, without
fine-tuning them all. Prior studies have revealed that well-trained models
exhibit the phenomenon of Neural Collapse. Based on a widely used neural
collapse metric in existing literature, we observe a strong correlation between
the neural collapse of pre-trained models and their corresponding fine-tuned
models. Inspired by this observation, we propose a novel method termed Fair
Collapse (FaCe) for transferability estimation by comprehensively measuring the
degree of neural collapse in the pre-trained model. Typically, FaCe comprises
two different terms: the variance collapse term, which assesses the class
separation and within-class compactness, and the class fairness term, which
quantifies the fairness of the pre-trained model towards each class. We
investigate FaCe on a variety of pre-trained classification models across
different network architectures, source datasets, and training loss functions.
Results show that FaCe yields state-of-the-art performance on different tasks
including image classification, semantic segmentation, and text classification,
which demonstrate the effectiveness and generalization of our method.