On the Dimensionality of Sentence Embeddings
Hongwei Wang Hongming Zhang Dong Yu
Learning sentence embeddings is a fundamental problem in natural language
processing. While existing research primarily focuses on enhancing the quality
of sentence embeddings, the exploration of sentence embedding dimensions is
limited. Here we present a comprehensive and empirical analysis of the
dimensionality of sentence embeddings. First, we demonstrate that the optimal
dimension of sentence embeddings is usually smaller than the default value.
Subsequently, to compress the dimension of sentence embeddings with minimum
performance degradation, we identify two components contributing to the overall
performance loss: the encoder's performance loss and the pooler's performance
loss. Therefore, we propose a two-step training method for sentence
representation learning models, wherein the encoder and the pooler are
optimized separately to mitigate the overall performance loss in low-dimension
scenarios. Experimental results on seven STS tasks and seven sentence
classification tasks demonstrate that our method significantly improves the
performance of low-dimensional sentence embeddings.