Quantization of a neural network has an inherent problem called accumulated quantization error, which is the key obstacle towards ultra-low precision, e.g., 2- or 3-bit precision. To resolve this problem, we propose precision highway, which forms an end-to-end high-precision information flow while performing the ultra-low-precision computation. First, we describe how the precision highway reduce the accumulated quantization error in both convolutional and recurrent neural networks. We also provide the quantitative analysis of the benefit of precision highway and evaluate the overhead on the state-of-the-art hardware accelerator. In the experiments, our proposed method outperforms the best existing quantization methods while offering 3-bit weight/activation quantization with no accuracy loss and 2-bit quantization with a 2.45 % top-1 accuracy loss in ResNet-50. We also report that the proposed method significantly outperforms the existing method in the 2-bit quantization of an LSTM for language modeling. Energy-efficient inference of neural networks is becoming increasingly important in both servers and mobile devices (e.g., smartphones, AR/VR devices, and drones). Recently, there have been active studies on ultra-low-precision inference using 1 to 4 bits BID21 BID9 BID27 BID25 BID28 BID2 BID15 BID1 and their implementations on CPU and GPU BID23 , and dedicated hardware BID19 BID22 . However, as will be explained in section 5.2, the existing quantization methods suffer from a problem called accumulated quantization error where large quantization errors get accumulated across layers, making it difficult to enable ultra-low precision in deep neural networks.In order to address this problem, we propose a novel concept called precision highway where an end-to-end path of high-precision information reduces the accumulated quantization error thereby enabling ultra-low-precision computation. Our proposed work is similar to recent studies BID15 BID2 which propose utilizing pre-activation residual networks, where skip connections are kept in full precision while the residual path performs low-precision computation. Compared with these works, our proposed method offers a generalized concept of high-precision information flow, namely, precision highway, which can be applied to not only the pre-activation convolutional networks but also both the post-activation convolutional and recurrent neural networks. Our contributions are as follows.• We propose a novel idea of network-level approach to quantization, called precision highway and quantitatively analyze its benefits in terms of the propagation of quantization errors and the difficulty of convergence in training based on the shape of loss surface.• We provide the detailed analysis of the energy and memory overhead of precision highway based on the state-of-the-art hardware accelerator model. According to our experiments, the overhead is negligible while offering significant improvements in accuracy.• We apply precision highway to both convolution and recurrent networks. We report a 3-bit quantization of ResNet-50 without accuracy loss and a 2-bit quantization with a very small accuracy loss. We also provide the sub 4-bit quantization results of long short-term memory (LSTM) for language modeling.2 RELATED WORK BID16 presented an int8 quantization method that selects an activation truncation threshold to minimize the Kullback-Leibler divergence between the distributions of the original and quantized data. BID12 proposed a quantization scheme that enables integer-arithmetic only matrix multiplications (practically, 8-bit quantization for neural networks). These methods are implemented on existing CPUs or GPUs BID11 ACL) . BID9 presented a binarization method and demonstrated the performance benefit on a GPU. BID21 proposed a binary network called XNOR-Net in which a weight-binarized AlexNet gives the same accuracy as a full-precision one. BID25 presented DoReFa-Net, which applies tanh-based weight quantization and bounded activation. BID26 proposed a balanced quantization that attempts to balance the population of values on quantization levels. BID7 proposed utilizing full precision for internal cell states in the LSTM because of their wide value distributions. This work is similar to ours in that high-precision data are selectively utilized to improve the quantized network. Our difference is proposing a network-level end-to-end flow of high-precision activation. Recently, BID28 presented 4-bit quantization with ResNet-50 . They adopt Dorefa-net style weight quantization with static bounded activation, and improve accuracy by adopting multi-step quantization and knowledge distillation during fine-tuning. proposed a trade-off between the number of channels and precision . Clustering-based methods have the potential to further reduce the precision BID5 . However, they require a lookup table and full-precision computation , which makes them less hardware-friendly.Recently, BID2 a) and BID15 proposed utilizing full-precision on the skip connections in pre-activation residual networks 1 . Compared with those works, our proposed idea has a salient difference in that it offers a network-level solution and demonstrates that the end-to-end flow of high-precision information is crucial. In addition, our method is not limited to pre-activation residual networks , but general enough to be applied to both post-activation convolutional and recurrent neural networks. In this paper, we proposed the concept of end-to-end precision highway which can be applied to both feedforward and feedback networks and enable ultra-low precision in deep neural networks. The proposed precision highway reduces quantization errors by keeping high-precision activation from the input to output of the network with small computation costs. We described how it reduces the accumulated quantization error and presented quantitative analyses in terms of accuracy and hardware cost as well as training characteristics. Our experiments showed that the proposed method outperforms the state-of-the-art methods in the 3-and 2-bit quantizations of ResNet-18/50 and 2-bit quantization of an LSTM model. We believe that our work will serve as a step toward mixed precision networks for computational efficiency.
precision highway; a generalized concept of high-precision information flow for sub 4-bit quantization  Investigates the problem of neural network quantization by employing an end-to-end precision highway to reduce the accumulated quantization error and enable ultra-low precision in deep neural networks.  This paper studies methods to improve the performance of quantized neural networks This paper proposes to keep a high activation/gradient flow in two kinds of networks structures, ResNet and LSTM.