FPTQ: Fine-grained Post-Training Quantization for Large Language Models
Qingyuan Li Yifan Zhang Liang Li Peng Yao Bo Zhang Xiangxiang Chu Yerui Sun Li Du Yuchen Xie
In the era of large-scale language models, the substantial parameter size
poses significant challenges for deployment. Being a prevalent compression
technique, quantization has emerged as the mainstream practice to tackle this
issue, which is mainly centered on two recipes W8A8 and W4A16 (i.e. weights and
activations in such bit widths). In this study, we propose a novel W4A8
post-training quantization method for the available open-sourced LLMs, which
combines the advantages of both two recipes. Therefore, we can leverage the
benefit in the I/O utilization of 4-bit weight quantization and the
acceleration due to 8-bit matrix computation. Nevertheless, the W4A8 faces
notorious performance degradation. As a remedy, we involve layerwise activation
quantization strategies which feature a novel logarithmic equalization for most
intractable layers, and we combine them with fine-grained weight quantization.
Without whistles and bells, we eliminate the necessity for further fine-tuning
and obtain the state-of-the-art W4A8 quantized performance on BLOOM, LLaMA, and
LLaMA-2 on standard benchmarks. We confirm that the W4A8 quantization is
achievable for the deployment of large language models, fostering their
wide-spreading real-world applications.