BatMan-CLR: Making Few-shots Meta-Learners Resilient Against Label Noise
Jeroen M. Galjaard Robert Birke Juan Perez Lydia Y. Chen
The negative impact of label noise is well studied in classical supervised
learning yet remains an open research question in meta-learning. Meta-learners
aim to adapt to unseen learning tasks by learning a good initial model in
meta-training and consecutively fine-tuning it according to new tasks during
meta-testing. In this paper, we present the first extensive analysis of the
impact of varying levels of label noise on the performance of state-of-the-art
meta-learners, specifically gradient-based $N$-way $K$-shot learners. We show
that the accuracy of Reptile, iMAML, and foMAML drops by up to 42% on the
Omniglot and CifarFS datasets when meta-training is affected by label noise. To
strengthen the resilience against label noise, we propose two sampling
techniques, namely manifold (Man) and batch manifold (BatMan), which transform
the noisy supervised learners into semi-supervised ones to increase the utility
of noisy labels. We first construct manifold samples of $N$-way
$2$-contrastive-shot tasks through augmentation, learning the embedding via a
contrastive loss in meta-training, and then perform classification through
zeroing on the embedding in meta-testing. We show that our approach can
effectively mitigate the impact of meta-training label noise. Even with 60%
wrong labels \batman and \man can limit the meta-testing accuracy drop to
${2.5}$, ${9.4}$, ${1.1}$ percent points, respectively, with existing
meta-learners across the Omniglot, CifarFS, and MiniImagenet datasets.