Length is a Curse and a Blessing for Document-level Semantics
Chenghao Xiao Yizhi Li G Thomas Hudson Chenghua Lin Noura Al Moubayed
In recent years, contrastive learning (CL) has been extensively utilized to
recover sentence and document-level encoding capability from pre-trained
language models. In this work, we question the length generalizability of
CL-based models, i.e., their vulnerability towards length-induced semantic
shift. We verify not only that length vulnerability is a significant yet
overlooked research gap, but we can devise unsupervised CL methods solely
depending on the semantic signal provided by document length. We first derive
the theoretical foundations underlying length attacks, showing that elongating
a document would intensify the high intra-document similarity that is already
brought by CL. Moreover, we found that isotropy promised by CL is highly
dependent on the length range of text exposed in training. Inspired by these
findings, we introduce a simple yet universal document representation learning
framework, LA(SER)$^{3}$: length-agnostic self-reference for semantically
robust sentence representation learning, achieving state-of-the-art
unsupervised performance on the standard information retrieval benchmark.