The tremendous memory and computational complexity of Convolutional Neural Networks (CNNs) prevents the inference deployment on resource-constrained systems. As a result, recent research focused on CNN optimization techniques, in particular quantization, which allows weights and activations of layers to be represented with just a few bits while achieving impressive prediction performance. However, aggressive quantization techniques still fail to achieve full-precision prediction performance on state-of-the-art CNN architectures on large-scale classification tasks. In this work we propose a method for weight and activation quantization that is scalable in terms of quantization levels (n-ary representations) and easy to compute while maintaining the performance close to full-precision CNNs. Our weight quantization scheme is based on trainable scaling factors and a nested-means clustering strategy which is robust to weight updates and therefore exhibits good convergence properties. The flexibility of nested-means clustering enables exploration of various n-ary weight representations with the potential of high parameter compression. For activations, we propose a linear quantization strategy that takes the statistical properties of batch normalization into account. We demonstrate the effectiveness of our approach using state-of-the-art models on ImageNet. The increasing computational complexity and memory requirements of Convolutional Neural Networks (CNNs) have motivated recent research efforts in efficient representation and processing of CNNs. Several optimization and inference approaches have been proposed with the objective of model compression and inference acceleration. The primary aim of model compression is to enable on-device storage (e.g., mobile phones and other resource-constrained devices) and to leverage on-chip memory in order to reduce energy consumption BID13 , latency and bandwidth of parameter accesses. Inference acceleration can be achieved by lowering the precision of computations in terms of resolution, removing connections within networks (pruning), and specialized software/hardware architectures.Quantized Neural Networks are optimization techniques where weights and/or activations of a neural network are transformed from 32-bit floating point into a lower resolution; for aggressive quantization techniques down to binary BID7 BID22 or ternary BID16 BID31 representations. Although prior quantization techniques achieve fullprecision accuracy on highly over-parameterized architectures (e.g., AlexNet, VGG) or toy tasks (e.g., MNIST, SVHN, CIFAR-10), there is still an unacceptable gap between extremely low bitwidth representations and state-of-the-art architectures for real-world tasks. Furthermore, aggressive quantization approaches are usually designed for specific representations (e.g., binary or ternary), and are not scalable in the sense that they do not allow for more quantization levels (i.e., weight values) if required. Thus, accuracy degradation cannot be compensated without changes in the baseline architecture which includes deepening or widening the neural network, and/or using full-precision weights in the input and output layers, respectively.In this work, we address the issue of accuracy degradation by introducing a scalable non-uniform quantization method for weights that is based on trainable scaling factors in combination with a nested-means clustering approach. In particular, nested-means splits the weight distribution iteratively into several quantization intervals until a pre-defined discretization level is reached. Subsequently, all weights within a certain quantization interval are assigned the same weight (i.e., the scaling factor). Nested-means clustering tends to assign small weights to larger quantization intervals while less frequent larger weights are assigned to smaller quantization intervals. This improves classification performance which is in line with recent observations that larger weights carry more information than smaller weights BID9 . We evaluate our approach on state-of-theart CNN architectures in terms of computational requirements and prediction accuracy using the ImageNet classification task.The paper is structured as follows. In Sec. 2 and Sec. 3 related work and background on inference acceleration is discussed. Weight and activation quantization is presented in Sec. 4 and Sec. 5, respectively. Experimental results for ImageNet are shown in Sec. 6. Sec. 7 concludes the paper. We have presented a novel approach for compressing CNNs through quantization and connection pruning, which reduces the resolution of weights and activations and is scalable in terms of the number of quantization levels. As a result, the computational complexity and memory requirements of DNNs are substantially reduced, and an execution on resource-constrained devices is more feasible. We introduced a nested-means clustering algorithm for weight quantization that finds suitable interval thresholds that are subsequently used to assign each weight to a trainable scaling factor. Our approach exhibits both a low computational complexity and robustness to weight updates, which makes it an attractive alternative to other clustering methods. Furthermore, the proposed quantization method is flexible as it allows for various numbers of quantization levels, enabling high compression rates while achieving prediction accuracies close to single-precision floating-point weights.For instance, we utilize this flexibility to add an extra quantization level to ternary weights (quaternary weights), resulting in an improvement in prediction accuracy while keeping the bit width at two. For activation quantization, we developed an approximation based on statistical attributes that have been observed when batch normalization is employed. Experiments using state-of-the-art DNN architectures on real-world tasks, including ResNet-18 and ImageNet, show the effectiveness of our approach.
We propose a quantization scheme for weights and activations of deep neural networks. This reduces the memory footprint substantially and accelerates inference. CNN model compression aand inference acceleration using quantization.