The state-of-the-art (SOTA) for mixed precision training is dominated by variants of low precision floating point operations, and in particular, FP16 accumulating into FP32 Micikevicius et al. (2017). On the other hand, while a lot of research has also happened in the domain of low and mixed-precision Integer training, these works either present results for non-SOTA networks (for instance only AlexNet for ImageNet-1K), or relatively small datasets (like CIFAR-10). In this work, we train state-of-the-art visual understanding neural networks on the ImageNet-1K dataset, with Integer operations on General Purpose (GP) hardware. In particular, we focus on Integer Fused-Multiply-and-Accumulate (FMA) operations which take two pairs of INT16 operands and accumulate results into an INT32 output.We propose a shared exponent representation of tensors and develop a Dynamic Fixed Point (DFP) scheme suitable for common neural network operations. The nuances of developing an efficient integer convolution kernel is examined, including methods to handle overflow of the INT32 accumulator. We implement CNN training for ResNet-50, GoogLeNet-v1, VGG-16 and AlexNet; and these networks achieve or exceed SOTA accuracy within the same number of iterations as their FP32 counterparts without any change in hyper-parameters and with a 1.8X improvement in end-to-end training throughput. To the best of our knowledge these results represent the first INT16 training results on GP hardware for ImageNet-1K dataset using SOTA CNNs and achieve highest reported accuracy using half precision While single precision floating point (FP32) representation has been the mainstay for deep learning training, half-precision and sub-half-precision arithmetic has recently captured interest of the academic and industrial research community. Primarily this interest stems from the ability to attain potentially upto 2X or more speedup of training as compared to FP32, when using half-precision fused-multiply and accumulate operations. For instance NVIDIA Volta NVIDIA (2017) provides 8X more half-precision Flops as compared to FP32.Unlike single precision floating point, which is a unanimous choice for 32b training, half-precision training can either use half-precision floating point (FP16), or integers (INT16). These two options offer varying degrees of precision and range; with INT16 having higher precision but lower dynamic range as compared to FP16. This also leads to residues between half-precision representation and single precision to be fundamentally different -with integer representations contributing lower residual errors for larger (and possibly more important) elements of a tensor. Beyond this first order distinction in data types, there are multiple algorithmic and semantic differences (for example FP16 multiply-and-accumulate operation accumulating into FP32 results) for each of these data types. Hence, when discussing half-precision training, the whole gamut of tensor representation, semantics of multiply-and-accumulate operation, down-conversion scheme (if the accumulation is to a higher precision), scaling and normalization techniques, and overflow management methods must be considered in totality to achieve SOTA accuracy. Indeed, unless the right combination of the aforesaid vectors are selected, half precision training is likely to fail. Conversely, drawing conclusions on the efficacy of a method by not selecting all vectors properly can lead to inaccurate conclusions.In this work we describe a mixed-precision training setup which uses:• INT16 tensors with shared tensor-wide exponent, with a potential to extend to sub-tensor wide exponents.• An instruction which multiplies two INT16 numbers and stores the output into a INT32 accumulator.• A down-convert scheme based on the maximum value of the output tensor in the current iteration using multiple rounding methods like nearest, stochastic, and biased rounding.• An overflow management scheme which accumulates partial INT32 results into FP32, along with trading off input precision with length of accumulate chain to gain performance.The compute for neural network training is dominated by GEMM-like, convolution, or dot-product operations. These are amenable to speedup via specialized low-precision instructions for fusedmultiply-and-accumulate (FMA), like AVX512_4VNNI 1 . However , this does not necessarily mean using half-precision representation for all tensors, or using only half-precision operations. In fact , performance speedups by migrating the compute intensive operations in both forward and back prorogation (FPROP, BPROP and WTGRAD) is often close to the maximum achievable speedup obtained by replacing all operations (for instance SGD) in half-precision. In cases where it is not, performance degradation typically happens due to limitations of memory bandwidth, and other architectural reasons.Hence on a balanced general purpose machine, a mixed-precision strategy of keeping precision critical operations (like SGD and some normalizations) in single precision and compute intensive operations in half precision can be employed. The proposed integer-16 based mixed-precision training follows this template.Using the aforesaid method, we train multiple visual understanding CNNs and achieve Top-1 accuracies BID16 on the ImageNet-1K dataset BID2 which match or exceed single precision results. These results are obtained without changing any hyper-parameters, and in as many iterations as the baseline FP32 training. We achieve 75.77 % Top-1 accuracy for ResNet-50 which, to the best of our knowledge, significantly exceeds any result published for halfprecision training, for example ; . Further, we also demonstrate our methodology achieves state-of-the-art accuracy (comparable to FP32 baseline) with int16 training on GoogLeNet-v1, VGG-16 and AlexNet networks. To the best of our knowledge, these are first such results using int16 training.The rest of the paper is organized as follows: Section 2 discusses the literature pertaining to various aspects of half-precision training. The dynamic fixed point format for representing half-precision tensors is described in Section 3. Dynamic fixed point kernels and neural network training operations are described in Section 4, and experimental results are presented in Section 5. Finally, we conclude this work in Section 6. We demonstrate industry-first reduced precision INT-based training result on large networks/data-sets. Showing on-par or better than FP32 baseline accuracies and potentially 2× savings in computation, communication and storage. Further, we propose a general dynamic fixed point representation scheme, with associated compute primitives and algorithm for the shared exponent management. This DFP solution can be used with general purpose hardware, leveraging the integer compute pipeline. We demonstrate this with implementation of CNN training for ResNet-50, GoogLeNet-v1, VGG-16 and AlexNet; training these networks with mixed precision DFP16 for the ImageNet-1K classification task. While this work focuses on visual understanding CNNs, in future we plan to demonstrate the efficacy of this method for other types of networks like RNNs, LSTMs, GANs and extend this to wider set of applications.
Mixed precision training pipeline using 16-bit integers on general purpose HW;  SOTA accuracy for ImageNet-class CNNs; Best reported accuracy for ImageNet-1K classification task with any reduced precision training; This paper shows that a careful implementation of mixed-precision dynamic fixed point computation can achieve state of the art accuracy using a reduced precision deep learning model with a 16 bit integer representation Proposes a "dynamic fixed point" scheme that shares the exponent part for a tensor and develops procedures to do NN computing with this format and demonstrates this for limited precision training.