Extensible Embedding: A Flexible Multipler For LLM's Context Length
Ninglu Shao Shitao Xiao Zheng Liu Peitian Zhang
Large language models (LLMs) call for extension of context to handle many
critical applications. However, the existing approaches are prone to expensive
costs and inferior quality of context extension. In this work, we propose
Extensible Embedding, which realizes high-quality extension of LLM's context
with strong flexibility and cost-effectiveness. Extensible embedding stand as
an enhancement of typical token embedding, which represents the information for
an extensible scope of context instead of a single token. By leveraging such
compact input units of higher information density, the LLM can access to a vast
scope of context even with a small context window. Extensible embedding is
systematically optimized in architecture and training method, which leads to
multiple advantages. 1) High flexibility of context extension, which flexibly
supports ad-hoc extension of diverse context lengths. 2) Strong sample
efficiency of training, which enables the embedding model to be learned in a
cost-effective way. 3) Superior compatibility with the existing LLMs, where the
extensible embedding can be seamlessly introduced as a plug-in component.
Comprehensive evaluations on long-context language modeling and understanding
tasks verify extensible embedding as an effective, efficient, flexible, and
compatible method to extend the LLM's context.