NUPES : Non-Uniform Post-Training Quantization via Power Exponent Search
Edouard Yvinec Arnaud Dapogny Kevin Bailly
Deep neural network (DNN) deployment has been confined to larger hardware
devices due to their expensive computational requirements. This challenge has
recently reached another scale with the emergence of large language models
(LLMs). In order to reduce both their memory footprint and latency, a promising
technique is quantization. It consists in converting floating point
representations to low bit-width fixed point representations, usually by
assuming a uniform mapping onto a regular grid. This process, referred to in
the literature as uniform quantization, may however be ill-suited as most DNN
weights and activations follow a bell-shaped distribution. This is even worse
on LLMs whose weight distributions are known to exhibit large, high impact,
outlier values. In this work, we propose an improvement over the most commonly
adopted way to tackle this limitation in deep learning models quantization,
namely, non-uniform quantization. NUPES leverages automorphisms to preserve the
scalar multiplications. Such transformations are derived from power functions.
However, the optimization of the exponent parameter and weight values remains a
challenging and novel problem which could not be solved with previous post
training optimization techniques which only learn to round up or down weight
values in order to preserve the predictive function. We circumvent this
limitation with a new paradigm: learning new quantized weights over the entire
quantized space. Similarly, we enable the optimization of the power exponent,
i.e. the optimization of the quantization operator itself during training by
alleviating all the numerical instabilities. The resulting predictive function
is compatible with integer-only low-bit inference. We show the ability of the
method to achieve state-of-the-art compression rates in both, data-free and
data-driven configurations.