Sparse Contrastive Learning of Sentence Embeddings
Ruize An Chen Zhang Dawei Song
Recently, SimCSE has shown the feasibility of contrastive learning in
training sentence embeddings and illustrates its expressiveness in spanning an
aligned and uniform embedding space. However, prior studies have shown that
dense models could contain harmful parameters that affect the model
performance, and it is no wonder that SimCSE can as well be invented with such
parameters. Driven by this, parameter sparsification is applied, where
alignment and uniformity scores are used to measure the contribution of each
parameter to the overall quality of sentence embeddings. Drawing from a
preliminary study, we consider parameters with minimal contributions to be
detrimental, as their sparsification results in improved model performance. To
discuss the ubiquity of detrimental parameters and remove them, more
experiments on the standard semantic textual similarity (STS) tasks and
transfer learning tasks are conducted, and the results show that the proposed
sparsified SimCSE (SparseCSE) has excellent performance in comparison with
SimCSE. Furthermore, through in-depth analysis, we establish the validity and
stability of our sparsification method, showcasing that the embedding space
generated by SparseCSE exhibits improved alignment compared to that produced by
SimCSE. Importantly, the uniformity yet remains uncompromised.