High throughput and low latency inference of deep neural networks are critical for the deployment of deep learning applications. This paper presents a general technique toward 8-bit low precision inference of convolutional neural networks, including 1) channel-wise scale factors of weights, especially for depthwise convolution, 2) Winograd convolution, and 3) topology-wise 8-bit support. We experiment the techniques on top of a widely-used deep learning framework. The 8-bit optimized model is automatically generated with a calibration process from FP32 model without the need of fine-tuning or retraining. We perform a systematical and comprehensive study on 18 widely-used convolutional neural networks and demonstrate the effectiveness of 8-bit low precision inference across a wide range of applications and use cases, including image classification, object detection, image segmentation, and super resolution. We show that the inference throughput
 and latency are improved by 1.6X and 1.5X respectively with minimal within 0.6%1to no loss in accuracy from FP32 baseline. We believe the methodology can provide the guidance and reference design of 8-bit low precision inference for other frameworks. All the code and models will be publicly available soon. While convolutional neural networks (CNN) shows state-of-the-art (SOTA) accuracy for wide range of computation vision tasks, it still faces challenges during industrial deployment due to its high computational complexity of inference. Low precision is one of the key techniques being actively studied recently to conquer the problem BID29 BID8 ; BID20 ; BID18 ; BID17 . With hardware acceleration support, low precision inference can compute more operations per second, reduce the memory access pressure and better utilize the cache, and deliver higher throughput and lower latency.Convolution is the primary operation in CNN models and it is a common practice to enable 8-bit low precision (INT8) inference for convolution in deep learning frameworks (e.g., TensorFlow, MXNet, and TensorRT). To make it work, convolution utilizes INT8 computation, which requires two scale factors for activation and weight, respectively. It is workable for standard convolution with single group and two groups BID13 . However, it does not work well for convolution with large groups, especially for depthwise convolution BID0 . In addition to direct convolution, it is worthwhile to explore INT8 Winograd convolution BID14 for better performance, which is absent in previous research 2 . Although recent work have demonstrated INT8 inference with minimal accuracy loss across various models BID29 BID4 ; ; BID11 , INT8 inference is limited due to more complex topology primarily introduced by sum operation in residual block and concatenation operation in inception block BID0 . Existing solutions need to convert the convolution output from INT8 to FP32, and apply the sum or concatenation operation on FP32. The sacrifice of memory bandwidth and frequent data conversion lead to considerable performance overhead and therefore limit the real deployment. Moreover, there is no systematical study of INT8 inference on various use cases, including image classification BID13 ; BID25 ; BID0 ; ), object detection Ren et al. (2015 ; BID1 ; BID15 , image segmentation BID16 ; BID15 , etc.In this paper, we present a general technique towards efficient INT8 inference of CNN models. We experiment the technique on top of a widely-used deep learning framework. To the best of our knowledge, our work is the first attempt to address the above problems. We summarize our contributions below:1. We provide a systematical approach to channel-wise quantization of convolution, which is essential to keep the accuracy for depthwise convolution. Top1 accuracy of INT8 inference on MobileNet-V1 and MobileNet-V2 is improved by 1.98% and 70.6%, respectively. 2. We explore the approach of INT8 Winograd convolution and present the calibration details that cannot be trivially derived from direct convolution. Our experiment on VGG-16 shows Top1 and Top5 accuracy loss with INT8 Winograd convolution is minimal within 0.30% and 0.25% from FP32 baseline, reducing from 5.31% and 3.38%, respectively. 3. We add the support of sum in residual block, concatenation in inception block, and convolution for classification. We also fuse the memory-bound operation convolution with a rectified linear unit (ReLU) BID19 and fold the parameters of batch normalization BID10 into convolution kernels. With topology-wise INT8 support, inference speed is greatly improved by data conversion reduction and memory saving. 4. To our knowledge, this is the first time such a systematic study is applied to and empirical result is reported on many CNN use cases and models. We develop a calibration tool that automatically generates optimized INT8 model from FP32 model without the need of fine-tuning or retraining for easy and repeatable deployment. We perform a comprehensive study on 18 widely-used CNN models and demonstrate the effectiveness of INT8 inference across a wide range of applications, including image classification, object detection, image segmentation, and super resolution. The inference throughput and latency are improved by 1.6X and 1.5X respectively, while the accuracy loss is minimal within 0.6% to no loss from FP32 baseline.We believe our methodology is general for CNN models and can provide the guide and reference on other frameworks. All the code and models will be publicly available soon.The rest of the paper is organized as follows, Section 2 discusses related work on low-precision inference in deep learning. Section 3 describes INT8 inference quantization approach and recipe for CNN models. Section 4 includes experimental results, comprehensive study, and related discussion. Finally, Section 5 concludes the summary with results and future directions. To align the model with best accuracy, the above performance in TAB2 does not include INT8 Winograd convolution. We expect to deliver similar performance improvement of Winograd on INT8 as FP32 BID14 during our development. Different from previous work BID29 BID26 , we also experiment the first convolution using INT8 than FP32, which shows reasonable accuracy within 1% loss.Our experimental results also demonstrate the impact of calibration process on accuracy with different sampling iteration, different calibration algorithm, or different scale factor mode. We summarize our findings: (1) Channel-wise scaling factors can always deliver better accuracy than single scale factor, especially for depthwise convolution; (2) Direct algorithm is more effective in most cases than KL, while KL algorithm can deliver better accuracy than FP32 baseline in some cases; and (3) More sampling iterations show more stable dynamic data rage and therefore better accuracy. How to select the optimal calibration strategy is an interesting topic as one of our future directions. In this paper, we propose the general recipe of INT8 inference and experiment the techniques on a widely-used deep learning framework. We develop an automatic calibration tool for optimized INT8 model generation and demonstrate the effectiveness on 18 CNN models across a wide range of use cases. The inference throughput and latency are improved by 1.6X and 1.5X respectively, while the accuracy loss is minimal within 0.6% to no loss from FP32 baseline. We believe our methodology is general for CNN models and can provide the guide and reference on other frameworks.
We present a general technique toward 8-bit low precision inference of convolutional neural networks.  This paper designs a system to automatically quantize the CNN pretrained models