HiCL: Hierarchical Contrastive Learning of Unsupervised Sentence
  Embeddings
Zhuofeng Wu Chaowei Xiao VG Vinod Vydiswaran
In this paper, we propose a hierarchical contrastive learning framework,
HiCL, which considers local segment-level and global sequence-level
relationships to improve training efficiency and effectiveness. Traditional
methods typically encode a sequence in its entirety for contrast with others,
often neglecting local representation learning, leading to challenges in
generalizing to shorter texts. Conversely, HiCL improves its effectiveness by
dividing the sequence into several segments and employing both local and global
contrastive learning to model segment-level and sequence-level relationships.
Further, considering the quadratic time complexity of transformers over input
tokens, HiCL boosts training efficiency by first encoding short segments and
then aggregating them to obtain the sequence representation. Extensive
experiments show that HiCL enhances the prior top-performing SNCSE model across
seven extensively evaluated STS tasks, with an average increase of +0.2%
observed on BERT-large and +0.44% on RoBERTa-large.