Neural network quantization has become an important research area due to its great impact on deployment of large models on resource constrained devices. In order to train networks that can be effectively discretized without loss of performance, we introduce a differentiable quantization procedure. Differentiability can be achieved by transforming continuous distributions over the weights and activations of the network to categorical distributions over the quantization grid. These are subsequently relaxed to continuous surrogates that can allow for efficient gradient-based optimization. We further show that stochastic rounding can be seen as a special case of the proposed approach and that under this formulation the quantization grid itself can also be optimized with gradient descent. We experimentally validate the performance of our method on MNIST, CIFAR 10 and Imagenet classification. Neural networks excel in a variety of large scale problems due to their highly flexible parametric nature. However, deploying big models on resource constrained devices, such as mobile phones, drones or IoT devices is still challenging because they require a large amount of power, memory and computation. Neural network compression is a means to tackle this issue and has therefore become an important research topic.Neural network compression can be, roughly, divided into two not mutually exclusive categories: pruning and quantization. While pruning BID18 BID10 aims to make the model "smaller" by altering the architecture, quantization aims to reduce the precision of the arithmetic operations in the network. In this paper we focus on the latter. Most network quantization methods either simulate or enforce discretization of the network during training, e.g. via rounding of the weights and activations. Although seemingly straighforward, the discontinuity of the discretization makes the gradient-based optimization infeasible. The reason is that there is no gradient of the loss with respect to the parameters. A workaround to the discontinuity are the "pseudo-gradients" according to the straight-through estimator BID3 , which have been successfully used for training low-bit width architectures at e.g. BID13 ; Zhu et al. (2016) .The purpose of this work is to introduce a novel quantization procedure, Relaxed Quantization (RQ). RQ can bypass the non-differentiability of the quantization operation during training by smoothing it appropriately. The contributions of this paper are four-fold: First, we show how to make the set of quantization targets part of the training process such that we can optimize them with gradient descent. Second , we introduce a way to discretize the network by converting distributions over the weights and activations to categorical distributions over the quantization grid. Third , we show that we can obtain a "smooth" quantization procedure by replacing the categorical distributions with (a) (b)Figure 1: The proposed discretization process. (a) Given a distribution p(x) over the real line we partition it into K intervals of width α where the center of each of the intervals is a grid point g i . The shaded area corresponds to the probability ofx falling inside the interval containing that specific g i .(b) Categorical distribution over the grid obtained after discretization. The probability of each of the grid points g i is equal to the probability ofx falling inside their respective intervals.concrete BID22 BID15 equivalents. Finally we show that stochastic rounding BID8 , one of the most popular quantization techniques, can be seen as a special case of the proposed framework. We present the details of our approach in Section 2, discuss related work in Section 3 and experimentally validate it in Section 4. Finally we conclude and provide fruitful directions for future research in Section 5. We have introduced Relaxed Quantization (RQ), a powerful and versatile algorithm for learning low-bit neural networks using a uniform quantization scheme. As such, the models trained by this method can be easily transferred and executed on low-bit fixed point chipsets. We have extensively evaluated RQ on various image classification benchmarks and have shown that it allows for the better trade-offs between accuracy and bit operations per second.Future hardware might enable us to cheaply do non-uniform quantization, for which this method can be easily extended. BID17 BID25 for example, show the benefits of low-bit floating point weights that can be efficiently implemented in hardware. The floating point quantization grid can be easily learned with RQ by redefiningĜ. General non-uniform quantization, as described TAB4 in the Appendix. We compare against multiple works that employ fixed-point quantization: SR+DR BID8 BID9 , LR Net BID29 , , TWN BID19 , INQ BID40 , BWN BID28 , XNORnet BID28 , DoReFa (Zhou et al., 2016) , HWGQ BID4 , ELQ Zhou et al. (2018) , SYQ BID7 , Apprentice , QSM BID30 and rounding.for example in BID2 , is a natural extension to RQ, whose exploration we leave to future work. For example, we could experiment with a base grid that is defined as in . Currently, the bit-width of every quantizer is determined beforehand, but in future work we will explore learning the required bit precision within this framework. In our experiments, batch normalization was implemented as a sequence of convolution, batch normalization and quantization. On a low-precision chip, however, batch normalization would be "folded" into the kernel and bias of the convolution, the result of which is then rounded to low precision. In order to accurately reflect this folding at test time, future work on the proposed algorithm will emulate folded batchnorm at training time and learn the corresponding quantization grid of the modified kernel and bias. For fast model evaluation on low-precision hardware, quantization goes hand-in-hand with network pruning. The proposed method is orthogonal to pruning methods such as, for example, L 0 regularization BID21 , which allows for group sparsity and pruning of hidden units.
We introduce a technique that allows for gradient based training of quantized neural networks. Proposes a unified and general way of training neural networks with reduced precision quantized synaptic weights and activations. A new approach to quantizing activations which is state of the art or competitive on several real image problems. A method for learning neural networks with quantized weights and activations by stochastically quantizing values and replacing the resulting categotical distribution with a continuous relaxation