Attendre: Wait To Attend By Retrieval With Evicted Queries in
  Memory-Based Transformers for Long Context Processing
Zi Yang Nan Hua
As LLMs have become capable of processing more complex types of inputs,
researchers have recently studied how to efficiently and affordably process
possibly arbitrarily long sequences. One effective approach is to use a FIFO
memory to store keys and values of an attention sublayer from past chunks to
allow subsequent queries to attend. However, this approach requires a large
memory and/or takes into the consideration the specific LM architecture.
Moreover, due to the causal nature between the key-values in prior context and
the queries at present, this approach cannot be extended to bidirectional
attention such as in an encoder-decoder or PrefixLM decoder-only architecture.
In this paper, we propose to use eviction policies, such as LRA and LFA, to
reduce the memory size and adapt to various architectures, and we also propose
the Attendre layer, a wait-to-attend mechanism by retrieving the key-value
memory (K/V memory) with evicted queries in the query memory (Q memory). As a
first step, we evaluate this method in the context length extension setup using
the TriviaQA reading comprehension task, and show the effectiveness of the
approach.