Understanding Transfer Learning and Gradient-Based Meta-Learning
  Techniques
Mike Huisman Aske Plaat Jan N. van Rijn
Deep neural networks can yield good performance on various tasks but often
require large amounts of data to train them. Meta-learning received
considerable attention as one approach to improve the generalization of these
networks from a limited amount of data. Whilst meta-learning techniques have
been observed to be successful at this in various scenarios, recent results
suggest that when evaluated on tasks from a different data distribution than
the one used for training, a baseline that simply finetunes a pre-trained
network may be more effective than more complicated meta-learning techniques
such as MAML, which is one of the most popular meta-learning techniques. This
is surprising as the learning behaviour of MAML mimics that of finetuning: both
rely on re-using learned features. We investigate the observed performance
differences between finetuning, MAML, and another meta-learning technique
called Reptile, and show that MAML and Reptile specialize for fast adaptation
in low-data regimes of similar data distribution as the one used for training.
Our findings show that both the output layer and the noisy training conditions
induced by data scarcity play important roles in facilitating this
specialization for MAML. Lastly, we show that the pre-trained features as
obtained by the finetuning baseline are more diverse and discriminative than
those learned by MAML and Reptile. Due to this lack of diversity and
distribution specialization, MAML and Reptile may fail to generalize to
out-of-distribution tasks whereas finetuning can fall back on the diversity of
the learned features.