Subspace Adaptation Prior for Few-Shot Learning
Mike Huisman Aske Plaat Jan N. van Rijn
Gradient-based meta-learning techniques aim to distill useful prior knowledge
from a set of training tasks such that new tasks can be learned more
efficiently with gradient descent. While these methods have achieved successes
in various scenarios, they commonly adapt all parameters of trainable layers
when learning new tasks. This neglects potentially more efficient learning
strategies for a given task distribution and may be susceptible to overfitting,
especially in few-shot learning where tasks must be learned from a limited
number of examples. To address these issues, we propose Subspace Adaptation
Prior (SAP), a novel gradient-based meta-learning algorithm that jointly learns
good initialization parameters (prior knowledge) and layer-wise parameter
subspaces in the form of operation subsets that should be adaptable. In this
way, SAP can learn which operation subsets to adjust with gradient descent
based on the underlying task distribution, simultaneously decreasing the risk
of overfitting when learning new tasks. We demonstrate that this ability is
helpful as SAP yields superior or competitive performance in few-shot image
classification settings (gains between 0.1% and 3.9% in accuracy). Analysis of
the learned subspaces demonstrates that low-dimensional operations often yield
high activation strengths, indicating that they may be important for achieving
good few-shot learning performance. For reproducibility purposes, we publish
all our research code publicly.