A Probabilistic Model to explain Self-Supervised Representation Learning
Alice Bizeul Bernhard Schölkopf Carl Allen
Self-supervised learning (SSL) learns representations by leveraging an
auxiliary unsupervised task, such as classifying semantically related samples,
e.g. different data augmentations or modalities. Of the many approaches to SSL,
contrastive methods, e.g. SimCLR, CLIP and VicREG, have gained attention for
learning representations that achieve downstream performance close to that of
supervised learning. However, a theoretical understanding of the mechanism
behind these methods eludes. We propose a generative latent variable model for
the data and show that several families of discriminative self-supervised
algorithms, including contrastive methods, approximately induce its latent
structure over representations, providing a unifying theoretical framework. We
also justify links to mutual information and the use of a projection head.
Fitting our model generatively, as SimVE, improves performance over previous
VAE methods on common benchmarks (e.g. FashionMNIST, CIFAR10, CelebA), narrows
the gap to discriminative methods on _content_ classification and, as our
analysis predicts, outperforms them where _style_ information is required,
taking a step toward task-agnostic representations.