Improving Generalization in Meta-Learning via Meta-Gradient Augmentation
Ren Wang Haoliang Sun Qi Wei Xiushan Nie Yuling Ma Yilong Yin
Meta-learning methods typically follow a two-loop framework, where each loop
potentially suffers from notorious overfitting, hindering rapid adaptation and
generalization to new tasks. Existing schemes solve it by enhancing the
mutual-exclusivity or diversity of training samples, but these data
manipulation strategies are data-dependent and insufficiently flexible. This
work alleviates overfitting in meta-learning from the perspective of gradient
regularization and proposes a data-independent \textbf{M}eta-\textbf{G}radient
\textbf{Aug}mentation (\textbf{MGAug}) method. The key idea is to first break
the rote memories by network pruning to address memorization overfitting in the
inner loop, and then the gradients of pruned sub-networks naturally form the
high-quality augmentation of the meta-gradient to alleviate learner overfitting
in the outer loop. Specifically, we explore three pruning strategies, including
\textit{random width pruning}, \textit{random parameter pruning}, and a newly
proposed \textit{catfish pruning} that measures a Meta-Memorization Carrying
Amount (MMCA) score for each parameter and prunes high-score ones to break rote
memories as much as possible. The proposed MGAug is theoretically guaranteed by
the generalization bound from the PAC-Bayes framework. In addition, we extend a
lightweight version, called MGAug-MaxUp, as a trade-off between performance
gains and resource overhead. Extensive experiments on multiple few-shot
learning benchmarks validate MGAug's effectiveness and significant improvement
over various meta-baselines. The code is publicly available at
\url{https://github.com/xxLifeLover/Meta-Gradient-Augmentation}.