Scaling Sentence Embeddings with Large Language Models
Ting Jiang Shaohan Huang Zhongzhi Luan Deqing Wang Fuzhen Zhuang
Large language models (LLMs) have recently garnered significant interest.
With in-context learning, LLMs achieve impressive results in various natural
language tasks. However, the application of LLMs to sentence embeddings remains
an area of ongoing research. In this work, we propose an in-context
learning-based method aimed at improving sentence embeddings performance. Our
approach involves adapting the previous prompt-based representation method for
autoregressive models, constructing a demonstration set that enables LLMs to
perform in-context learning, and scaling up the LLMs to different model sizes.
Through extensive experiments, in-context learning enables LLMs to generate
high-quality sentence embeddings without any fine-tuning. It helps LLMs achieve
performance comparable to current contrastive learning methods. By scaling
model size, we find scaling to more than tens of billion parameters harms the
performance on semantic textual similarity (STS) tasks. However, the largest
model outperforms other counterparts and achieves the new state-of-the-art
result on transfer tasks. We also fine-tune LLMs with current contrastive
learning approach, and the 2.7B OPT model, incorporating our prompt-based
method, surpasses the performance of 4.8B ST5, achieving the new
state-of-the-art results on STS tasks. Our code is available at
https://github.com/kongds/scaling_sentemb.