Pre-Trained Model Recommendation for Downstream Fine-tuning
Jiameng Bai Sai Wu Jie Song Junbo Zhao Gang Chen
As a fundamental problem in transfer learning, model selection aims to rank
off-the-shelf pre-trained models and select the most suitable one for the new
target task. Existing model selection techniques are often constrained in their
scope and tend to overlook the nuanced relationships between models and tasks.
In this paper, we present a pragmatic framework \textbf{Fennec}, delving into a
diverse, large-scale model repository while meticulously considering the
intricate connections between tasks and models. The key insight is to map all
models and historical tasks into a transfer-related subspace, where the
distance between model vectors and task vectors represents the magnitude of
transferability. A large vision model, as a proxy, infers a new task's
representation in the transfer space, thereby circumventing the computational
burden of extensive forward passes. We also investigate the impact of the
inherent inductive bias of models on transfer results and propose a novel
method called \textbf{archi2vec} to encode the intricate structures of models.
The transfer score is computed through straightforward vector arithmetic with a
time complexity of $\mathcal{O}(1)$. Finally, we make a substantial
contribution to the field by releasing a comprehensive benchmark. We validate
the effectiveness of our framework through rigorous testing on two benchmarks.
The benchmark and the code will be publicly available in the near future.