IntactKV: Improving Large Language Model Quantization by Keeping Pivot
  Tokens Intact
Ruikang Liu Haoli Bai Haokun Lin Yuening Li Han Gao Zhengzhuo Xu Lu Hou Jun Yao Chun Yuan
Large language models (LLMs) excel in natural language processing but demand
intensive computation. To mitigate this, various quantization methods have been
explored, yet they compromise LLM performance. This paper unveils a previously
overlooked type of outlier in LLMs. Such outliers are found to allocate most of
the attention scores on initial tokens of input, termed as pivot tokens, which
is crucial to the performance of quantized LLMs. Given that, we propose
IntactKV to generate the KV cache of pivot tokens losslessly from the
full-precision model. The approach is simple and easy to combine with existing
quantization solutions. Besides, IntactKV can be calibrated as additional LLM
parameters to boost the quantized LLMs further. Mathematical analysis also
proves that IntactKV effectively reduces the upper bound of quantization error.
Empirical results show that IntactKV brings consistent improvement and achieves
lossless weight-only INT4 quantization on various downstream tasks, leading to
the new state-of-the-art for LLM quantization.