Effect of Weight Quantization on Learning Models by Typical Case
  Analysis
Shuhei Kashiwamura Ayaka Sakata Masaaki Imaizumi
This paper examines the quantization methods used in large-scale data
analysis models and their hyperparameter choices. The recent surge in data
analysis scale has significantly increased computational resource requirements.
To address this, quantizing model weights has become a prevalent practice in
data analysis applications such as deep learning. Quantization is particularly
vital for deploying large models on devices with limited computational
resources. However, the selection of quantization hyperparameters, like the
number of bits and value range for weight quantization, remains an
underexplored area. In this study, we employ the typical case analysis from
statistical physics, specifically the replica method, to explore the impact of
hyperparameters on the quantization of simple learning models. Our analysis
yields three key findings: (i) an unstable hyperparameter phase, known as
replica symmetry breaking, occurs with a small number of bits and a large
quantization width; (ii) there is an optimal quantization width that minimizes
error; and (iii) quantization delays the onset of overparameterization, helping
to mitigate overfitting as indicated by the double descent phenomenon. We also
discover that non-uniform quantization can enhance stability. Additionally, we
develop an approximate message-passing algorithm to validate our theoretical
results.