Towards Better Understanding of Contrastive Sentence Representation
  Learning: A Unified Paradigm for Gradient
Mingxin Li Richong Zhang Zhijie Nie
Sentence Representation Learning (SRL) is a crucial task in Natural Language
Processing (NLP), where contrastive Self-Supervised Learning (SSL) is currently
a mainstream approach. However, the reasons behind its remarkable effectiveness
remain unclear. Specifically, in other research fields, contrastive SSL shares
similarities in both theory and practical performance with non-contrastive SSL
(e.g., alignment & uniformity, Barlow Twins, and VICReg). However, in SRL,
contrastive SSL outperforms non-contrastive SSL significantly. Therefore, two
questions arise: First, what commonalities enable various contrastive losses to
achieve superior performance in SRL? Second, how can we make non-contrastive
SSL, which is similar to contrastive SSL but ineffective in SRL, effective? To
address these questions, we start from the perspective of gradients and
discover that four effective contrastive losses can be integrated into a
unified paradigm, which depends on three components: the Gradient Dissipation,
the Weight, and the Ratio. Then, we conduct an in-depth analysis of the roles
these components play in optimization and experimentally demonstrate their
significance for model performance. Finally, by adjusting these components, we
enable non-contrastive SSL to achieve outstanding performance in SRL.