A2Q+: Improving Accumulator-Aware Weight Quantization
Ian Colbert Alessandro Pappalardo Jakoba Petri-Koenig Yaman Umuroglu
Quantization techniques commonly reduce the inference costs of neural
networks by restricting the precision of weights and activations. Recent
studies show that also reducing the precision of the accumulator can further
improve hardware efficiency at the risk of numerical overflow, which introduces
arithmetic errors that can degrade model accuracy. To avoid numerical overflow
while maintaining accuracy, recent work proposed accumulator-aware quantization
(A2Q), a quantization-aware training method that constrains model weights
during training to safely use a target accumulator bit width during inference.
Although this shows promise, we demonstrate that A2Q relies on an overly
restrictive constraint and a sub-optimal weight initialization strategy that
each introduce superfluous quantization error. To address these shortcomings,
we introduce: (1) an improved bound that alleviates accumulator constraints
without compromising overflow avoidance; and (2) a new strategy for
initializing quantized weights from pre-trained floating-point checkpoints. We
combine these contributions with weight normalization to introduce A2Q+. We
support our analysis with experiments that show A2Q+ significantly improves the
trade-off between accumulator bit width and model accuracy and characterize new
trade-offs that arise as a consequence of accumulator constraints.