The What, Why, and How of Context Length Extension Techniques in Large
  Language Models -- A Detailed Survey
Saurav Pawar S. M Towhidul Islam Tonmoy S M Mehedi Zaman Vinija Jain Aman Chadha Amitava Das
The advent of Large Language Models (LLMs) represents a notable breakthrough
in Natural Language Processing (NLP), contributing to substantial progress in
both text comprehension and generation. However, amidst these advancements, it
is noteworthy that LLMs often face a limitation in terms of context length
extrapolation. Understanding and extending the context length for LLMs is
crucial in enhancing their performance across various NLP applications. In this
survey paper, we delve into the multifaceted aspects of exploring why it is
essential, and the potential transformations that superior techniques could
bring to NLP applications. We study the inherent challenges associated with
extending context length and present an organized overview of the existing
strategies employed by researchers. Additionally, we discuss the intricacies of
evaluating context extension techniques and highlight the open challenges that
researchers face in this domain. Furthermore, we explore whether there is a
consensus within the research community regarding evaluation standards and
identify areas where further agreement is needed. This comprehensive survey
aims to serve as a valuable resource for researchers, guiding them through the
nuances of context length extension techniques and fostering discussions on
future advancements in this evolving field.