The Common Stability Mechanism behind most Self-Supervised Learning
  Approaches
Abhishek Jha Matthew B. Blaschko Yuki M. Asano Tinne Tuytelaars
Last couple of years have witnessed a tremendous progress in self-supervised
learning (SSL), the success of which can be attributed to the introduction of
useful inductive biases in the learning process to learn meaningful visual
representations while avoiding collapse. These inductive biases and constraints
manifest themselves in the form of different optimization formulations in the
SSL techniques, e.g. by utilizing negative examples in a contrastive
formulation, or exponential moving average and predictor in BYOL and SimSiam.
In this paper, we provide a framework to explain the stability mechanism of
these different SSL techniques: i) we discuss the working mechanism of
contrastive techniques like SimCLR, non-contrastive techniques like BYOL, SWAV,
SimSiam, Barlow Twins, and DINO; ii) we provide an argument that despite
different formulations these methods implicitly optimize a similar objective
function, i.e. minimizing the magnitude of the expected representation over all
data samples, or the mean of the data distribution, while maximizing the
magnitude of the expected representation of individual samples over different
data augmentations; iii) we provide mathematical and empirical evidence to
support our framework. We formulate different hypotheses and test them using
the Imagenet100 dataset.