Is Pre-training Truly Better Than Meta-Learning?
Brando Miranda Patrick Yu Saumya Goyal Yu-Xiong Wang Sanmi Koyejo
In the context of few-shot learning, it is currently believed that a fixed
pre-trained (PT) model, along with fine-tuning the final layer during
evaluation, outperforms standard meta-learning algorithms. We re-evaluate these
claims under an in-depth empirical examination of an extensive set of formally
diverse datasets and compare PT to Model Agnostic Meta-Learning (MAML). Unlike
previous work, we emphasize a fair comparison by using: the same architecture,
the same optimizer, and all models trained to convergence. Crucially, we use a
more rigorous statistical tool -- the effect size (Cohen's d) -- to determine
the practical significance of the difference between a model trained with PT
vs. a MAML. We then use a previously proposed metric -- the diversity
coefficient -- to compute the average formal diversity of a dataset. Using this
analysis, we demonstrate the following: 1. when the formal diversity of a data
set is low, PT beats MAML on average and 2. when the formal diversity is high,
MAML beats PT on average. The caveat is that the magnitude of the average
difference between a PT vs. MAML using the effect size is low (according to
classical statistical thresholds) -- less than 0.2. Nevertheless, this
observation is contrary to the currently held belief that a pre-trained model
is always better than a meta-learning model. Our extensive experiments consider
21 few-shot learning benchmarks, including the large-scale few-shot learning
dataset Meta-Data set. We also show no significant difference between a MAML
model vs. a PT model with GPT-2 on Openwebtext. We, therefore, conclude that a
pre-trained model does not always beat a meta-learned model and that the formal
diversity of a dataset is a driving factor.