CMLM-CSE: Based on Conditional MLM Contrastive Learning for Sentence
  Embeddings
Wei Zhang Xu Chen
Traditional comparative learning sentence embedding directly uses the encoder
to extract sentence features, and then passes in the comparative loss function
for learning. However, this method pays too much attention to the sentence body
and ignores the influence of some words in the sentence on the sentence
semantics. To this end, we propose CMLM-CSE, an unsupervised contrastive
learning framework based on conditional MLM. On the basis of traditional
contrastive learning, an additional auxiliary network is added to integrate
sentence embedding to perform MLM tasks, forcing sentence embedding to learn
more masked word information. Finally, when Bertbase was used as the
pretraining language model, we exceeded SimCSE by 0.55 percentage points on
average in textual similarity tasks, and when Robertabase was used as the
pretraining language model, we exceeded SimCSE by 0.3 percentage points on
average in textual similarity tasks.