Unsupervised Sentence Representation Learning with Frequency-induced
  Adversarial Tuning and Incomplete Sentence Filtering
Bing Wang Ximing Li Zhiyao Yang Yuanyuan Guan Jiayin Li Shengsheng Wang
Pre-trained Language Model (PLM) is nowadays the mainstay of Unsupervised
Sentence Representation Learning (USRL). However, PLMs are sensitive to the
frequency information of words from their pre-training corpora, resulting in
anisotropic embedding space, where the embeddings of high-frequency words are
clustered but those of low-frequency words disperse sparsely. This anisotropic
phenomenon results in two problems of similarity bias and information bias,
lowering the quality of sentence embeddings. To solve the problems, we
fine-tune PLMs by leveraging the frequency information of words and propose a
novel USRL framework, namely Sentence Representation Learning with
Frequency-induced Adversarial tuning and Incomplete sentence filtering
(SLT-FAI). We calculate the word frequencies over the pre-training corpora of
PLMs and assign words thresholding frequency labels. With them, (1) we
incorporate a similarity discriminator used to distinguish the embeddings of
high-frequency and low-frequency words, and adversarially tune the PLM with it,
enabling to achieve uniformly frequency-invariant embedding space; and (2) we
propose a novel incomplete sentence detection task, where we incorporate an
information discriminator to distinguish the embeddings of original sentences
and incomplete sentences by randomly masking several low-frequency words,
enabling to emphasize the more informative low-frequency words. Our SLT-FAI is
a flexible and plug-and-play framework, and it can be integrated with existing
USRL techniques. We evaluate SLT-FAI with various backbones on benchmark
datasets. Empirical results indicate that SLT-FAI can be superior to the
existing USRL baselines. Our code is released in
\url{https://github.com/wangbing1416/SLT-FAI}.