Nugget 2D: Dynamic Contextual Compression for Scaling Decoder-only
  Language Models
Guanghui Qin Corby Rosset Ethan C. Chau Nikhil Rao Benjamin Van Durme
Standard Transformer-based language models (LMs) scale poorly to long
contexts. We propose a solution based on dynamic contextual compression, which
extends the Nugget approach of Qin & Van Durme (2023) from BERT-like frameworks
to decoder-only LMs. Our method models history as compressed "nuggets" which
are trained to allow for reconstruction, and it can be initialized with
off-the-shelf models such as LLaMA. We demonstrate through experiments in
language modeling, question answering, and summarization that Nugget2D retains
capabilities in these tasks, while drastically reducing the overhead during
decoding in terms of time and space. For example, in the experiments of
autoencoding, Nugget2D can shrink context at a 20x compression ratio with a
BLEU score of 98% for reconstruction, achieving nearly lossless encoding.