Probabilistic Self-supervised Learning via Scoring Rules Minimization
Amirhossein Vahidi Simon Schoßer Lisa Wimmer Yawei Li Bernd Bischl Eyke Hüllermeier Mina Rezaei
In this paper, we propose a novel probabilistic self-supervised learning via
Scoring Rule Minimization (ProSMIN), which leverages the power of probabilistic
models to enhance representation quality and mitigate collapsing
representations. Our proposed approach involves two neural networks; the online
network and the target network, which collaborate and learn the diverse
distribution of representations from each other through knowledge distillation.
By presenting the input samples in two augmented formats, the online network is
trained to predict the target network representation of the same sample under a
different augmented view. The two networks are trained via our new loss
function based on proper scoring rules. We provide a theoretical justification
for ProSMIN's convergence, demonstrating the strict propriety of its modified
scoring rule. This insight validates the method's optimization process and
contributes to its robustness and effectiveness in improving representation
quality. We evaluate our probabilistic model on various downstream tasks, such
as in-distribution generalization, out-of-distribution detection, dataset
corruption, low-shot learning, and transfer learning. Our method achieves
superior accuracy and calibration, surpassing the self-supervised baseline in a
wide range of experiments on large-scale datasets like ImageNet-O and
ImageNet-C, ProSMIN demonstrates its scalability and real-world applicability.