Simple Techniques for Enhancing Sentence Embeddings in Generative
  Language Models
Bowen Zhang Kehua Chang Chunping Li
Sentence Embedding stands as a fundamental task within the realm of Natural
Language Processing, finding extensive application in search engines, expert
systems, and question-and-answer platforms. With the continuous evolution of
large language models such as LLaMA and Mistral, research on sentence embedding
has recently achieved notable breakthroughs. However, these advancements mainly
pertain to fine-tuning scenarios, leaving explorations into computationally
efficient direct inference methods for sentence representation in a nascent
stage. This paper endeavors to bridge this research gap. Through comprehensive
experimentation, we challenge the widely held belief in the necessity of an
Explicit One-word Limitation for deriving sentence embeddings from Pre-trained
Language Models (PLMs). We demonstrate that this approach, while beneficial for
generative models under direct inference scenario, is not imperative for
discriminative models or the fine-tuning of generative PLMs. This discovery
sheds new light on the design of manual templates in future studies. Building
upon this insight, we propose two innovative prompt engineering techniques
capable of further enhancing the expressive power of PLMs' raw embeddings:
Pretended Chain of Thought and Knowledge Enhancement. We confirm their
effectiveness across various PLM types and provide a detailed exploration of
the underlying factors contributing to their success.