A Probabilistic Approach to Self-Supervised Learning using Cyclical
  Stochastic Gradient MCMC
Masoumeh Javanbakhat Christoph Lippert
In this paper we present a practical Bayesian self-supervised learning method
with Cyclical Stochastic Gradient Hamiltonian Monte Carlo (cSGHMC). Within this
framework, we place a prior over the parameters of a self-supervised learning
model and use cSGHMC to approximate the high dimensional and multimodal
posterior distribution over the embeddings. By exploring an expressive
posterior over the embeddings, Bayesian self-supervised learning produces
interpretable and diverse representations. Marginalizing over these
representations yields a significant gain in performance, calibration and
out-of-distribution detection on a variety of downstream classification tasks.
We provide experimental results on multiple classification tasks on four
challenging datasets. Moreover, we demonstrate the effectiveness of the
proposed method in out-of-distribution detection using the SVHN and CIFAR-10
datasets.