PIPE : Parallelized Inference Through Post-Training Quantization
  Ensembling of Residual Expansions
Edouard Yvinec Arnaud Dapogny Kevin Bailly
Deep neural networks (DNNs) are ubiquitous in computer vision and natural
language processing, but suffer from high inference cost. This problem can be
addressed by quantization, which consists in converting floating point
perations into a lower bit-width format. With the growing concerns on privacy
rights, we focus our efforts on data-free methods. However, such techniques
suffer from their lack of adaptability to the target devices, as a hardware
typically only support specific bit widths. Thus, to adapt to a variety of
devices, a quantization method shall be flexible enough to find good accuracy
v.s. speed trade-offs for every bit width and target device. To achieve this,
we propose PIPE, a quantization method that leverages residual error expansion,
along with group sparsity and an ensemble approximation for better
parallelization. PIPE is backed off by strong theoretical guarantees and
achieves superior performance on every benchmarked application (from vision to
NLP tasks), architecture (ConvNets, transformers) and bit-width (from int8 to
ternary quantization).