Cherry on Top: Parameter Heterogeneity and Quantization in Large
  Language Models
Wanyun Cui Qianle Wang
This paper reveals the phenomenon of parameter heterogeneity in large
language models (LLMs). We find that a small subset of ``cherry'' parameters
exhibit a disproportionately large influence on model performance, while the
vast majority of parameters have minimal impact. This heterogeneity is found to
be prevalent across different model families, scales, and types. Motivated by
this observation, we propose CherryQ, a novel quantization method that unifies
the optimization of mixed-precision parameters. CherryQ identifies and
preserves the critical cherry parameters in high precision while aggressively
quantizing the remaining parameters to low precision. Extensive experiments
demonstrate the effectiveness of CherryQ. CherryQ outperforms existing
quantization approaches in terms of perplexity and downstream task performance.
Notably, our 3-bit quantized Vicuna-1.5 exhibits competitive performance
compared to their 16-bit counterparts. These findings highlight the potential
of CherryQ for enabling efficient deployment of LLMs by taking advantage of
parameter heterogeneity.