Unleash Model Potential: Bootstrapped Meta Self-supervised Learning
Jingyao Wang Zeen Song Wenwen Qiang Changwen Zheng
The long-term goal of machine learning is to learn general visual
representations from a small amount of data without supervision, mimicking
three advantages of human cognition: i) no need for labels, ii) robustness to
data scarcity, and iii) learning from experience. Self-supervised learning and
meta-learning are two promising techniques to achieve this goal, but they both
only partially capture the advantages and fail to address all the problems.
Self-supervised learning struggles to overcome the drawbacks of data scarcity,
while ignoring prior knowledge that can facilitate learning and generalization.
Meta-learning relies on supervised information and suffers from a bottleneck of
insufficient learning. To address these issues, we propose a novel Bootstrapped
Meta Self-Supervised Learning (BMSSL) framework that aims to simulate the human
learning process. We first analyze the close relationship between meta-learning
and self-supervised learning. Based on this insight, we reconstruct tasks to
leverage the strengths of both paradigms, achieving advantages i and ii.
Moreover, we employ a bi-level optimization framework that alternates between
solving specific tasks with a learned ability (first level) and improving this
ability (second level), attaining advantage iii. To fully harness its power, we
introduce a bootstrapped target based on meta-gradient to make the model its
own teacher. We validate the effectiveness of our approach with comprehensive
theoretical and empirical study.