Towards Few-Shot Adaptation of Foundation Models via Multitask
  Finetuning
Zhuoyan Xu Zhenmei Shi Junyi Wei Fangzhou Mu Yin Li Yingyu Liang
Foundation models have emerged as a powerful tool for many AI problems.
Despite the tremendous success of foundation models, effective adaptation to
new tasks, particularly those with limited labels, remains an open question and
lacks theoretical understanding. An emerging solution with recent success in
vision and NLP involves finetuning a foundation model on a selection of
relevant tasks, before its adaptation to a target task with limited labeled
samples. In this paper, we study the theoretical justification of this
multitask finetuning approach. Our theoretical analysis reveals that with a
diverse set of related tasks, this multitask finetuning leads to reduced error
in the target task, in comparison to directly adapting the same pretrained
model. We quantify the relationship between finetuning tasks and target tasks
by diversity and consistency metrics, and further propose a practical task
selection algorithm. We substantiate our theoretical claims with extensive
empirical evidence. Further, we present results affirming our task selection
algorithm adeptly chooses related finetuning tasks, providing advantages to the
model performance on target tasks. We believe our study shed new light on the
effective adaptation of foundation models to new tasks that lack abundant
labels. Our code is available at
https://github.com/OliverXUZY/Foudation-Model_Multitask.