MixQuant: Mixed Precision Quantization with a Bit-width Optimization
  Search
Eliska Kloberdanz Wei Le
Quantization is a technique for creating efficient Deep Neural Networks
(DNNs), which involves performing computations and storing tensors at lower
bit-widths than f32 floating point precision. Quantization reduces model size
and inference latency, and therefore allows for DNNs to be deployed on
platforms with constrained computational resources and real-time systems.
However, quantization can lead to numerical instability caused by roundoff
error which leads to inaccurate computations and therefore, a decrease in
quantized model accuracy. Similarly to prior works, which have shown that both
biases and activations are more sensitive to quantization and are best kept in
full precision or quantized with higher bit-widths, we show that some weights
are more sensitive than others which should be reflected on their quantization
bit-width. To that end we propose MixQuant, a search algorithm that finds the
optimal custom quantization bit-width for each layer weight based on roundoff
error and can be combined with any quantization method as a form of
pre-processing optimization. We show that combining MixQuant with BRECQ, a
state-of-the-art quantization method, yields better quantized model accuracy
than BRECQ alone. Additionally, we combine MixQuant with vanilla asymmetric
quantization to show that MixQuant has the potential to optimize the
performance of any quantization technique.