Exploring Model Transferability through the Lens of Potential Energy
Xiaotong Li Zixuan Hu Yixiao Ge Ying Shan Ling-Yu Duan
Transfer learning has become crucial in computer vision tasks due to the vast
availability of pre-trained deep learning models. However, selecting the
optimal pre-trained model from a diverse pool for a specific downstream task
remains a challenge. Existing methods for measuring the transferability of
pre-trained models rely on statistical correlations between encoded static
features and task labels, but they overlook the impact of underlying
representation dynamics during fine-tuning, leading to unreliable results,
especially for self-supervised models. In this paper, we present an insightful
physics-inspired approach named PED to address these challenges. We reframe the
challenge of model selection through the lens of potential energy and directly
model the interaction forces that influence fine-tuning dynamics. By capturing
the motion of dynamic representations to decline the potential energy within a
force-driven physical model, we can acquire an enhanced and more stable
observation for estimating transferability. The experimental results on 10
downstream tasks and 12 self-supervised models demonstrate that our approach
can seamlessly integrate into existing ranking techniques and enhance their
performances, revealing its effectiveness for the model selection task and its
potential for understanding the mechanism in transfer learning. Code will be
available at https://github.com/lixiaotong97/PED.