Non-contrastive sentence representations via self-supervision
Marco Farina Duccio Pappadopulo
Sample contrastive methods, typically referred to simply as contrastive are
the foundation of most unsupervised methods to learn text and sentence
embeddings. On the other hand, a different class of self-supervised loss
functions and methods have been considered in the computer vision community and
referred to as dimension contrastive. In this paper, we thoroughly compare this
class of methods with the standard baseline for contrastive sentence
embeddings, SimCSE. We find that self-supervised embeddings trained using
dimension contrastive objectives can outperform SimCSE on downstream tasks
without needing auxiliary loss functions.