Recurrent neural networks have achieved excellent performance in many applications. However, on portable devices with limited resources, the models are often too large to deploy. For applications on the server with large scale concurrent requests, the latency during inference can also be very critical for costly computing resources. In this work, we address these problems by quantizing the network, both weights and activations, into multiple binary codes {-1,+1}. We formulate the quantization as an optimization problem. Under the key observation that once the quantization coefficients are fixed the binary codes can be derived efficiently by binary search tree, alternating minimization is then applied.   We test the quantization for two well-known RNNs, i.e., long short term memory (LSTM) and gated recurrent unit (GRU), on the language models. Compared with the full-precision counter part, by 2-bit quantization we can achieve ~16x memory saving and  ~6x real inference acceleration on CPUs, with only a reasonable loss in the accuracy. By 3-bit quantization, we can achieve almost no loss in the accuracy or even surpass the original model, with ~10.5x memory saving and ~3x real inference acceleration. Both results beat the exiting quantization works with large margins.   We extend our alternating quantization to image classification tasks. In both RNNs and feedforward neural networks, the method also achieves  excellent performance. Recurrent neural networks (RNNs) are specific type of neural networks which are designed to model the sequence data. In last decades, various RNN architectures have been proposed, such as LongShort-Term Memory (LSTM) BID9 and Gated Recurrent Units BID0 . They have enabled the RNNs to achieve state-of-art performance in many applications, e.g., language models (Mikolov et al., 2010) , neural machine translation BID21 , automatic speech recognition BID5 , image captions BID23 , etc. However, the models often build on high dimensional input/output,e.g., large vocabulary in language models, or very deep inner recurrent networks, making the models have too many parameters to deploy on portable devices with limited resources. In addition, RNNs can only be executed sequentially with dependence on current hidden states. This causes large latency during inference. For applications in the server with large scale concurrent requests, e.g., on-line machine translation and speech recognition, large latency leads to limited requests processed per machine to meet the stringent response time requirements. Thus much more costly computing resources are in demand for RNN based models.To alleviate the above problems, several techniques can be employed, i.e., low rank approximation (Sainath et al., 2013; BID14 BID16 BID22 , sparsity BID19 BID7 , and quantization. All of them are build on the redundancy of current networks and can be combined. In this work, we mainly focus on quantization based methods. More precisely, we are to quantize all parameters into multiple binary codes {−1, +1}.The idea of quantizing both weights and activations is firstly proposed by BID11 . It has shown that even 1-bit binarization can achieve reasonably good performance in some visual classification tasks. Compared with the full precision counterpart, binary weights reduce the memory by a factor of 32. And the costly arithmetic operations between weights and activations can then be replaced by cheap XNOR and bitcount operations BID11 , which potentially leads to much acceleration. Rastegari et al. (2016) further incorporate a real coefficient to compensate for the binarization error. They apply the method to the challenging ImageNet dataset and achieve better performance than pure binarization in BID11 . However, it is still of large gap compared with the full precision networks. To bridge this gap, some recent works BID12 BID29 further employ quantization with more bits and achieve plausible performance. Meanwhile, quite an amount of works, e.g., BID3 BID30 BID6 , quantize the weights only. Although much memory saving can be achieved, the acceleration is very limited in modern computing devices (Rastegari et al., 2016) .Among all existing quantization works, most of them focus on convolutional neural networks (CNNs) while pay less attention to RNNs. As mentioned earlier, the latter is also very demanding. Recently, BID10 showed that binarized LSTM with preconditioned coefficients can achieve promising performance in some easy tasks such as predicting the next character. However, for RNNs with large input/output , e.g., large vocabulary in language models, it is still very challenging for quantization. Both works of BID12 and BID28 test the effectiveness of their multi-bit quantized RNNs to predict the next word. Although using up to 4-bits, the results with quantization still have noticeable gap with those with full precision. This motivates us to find a better method to quantize RNNs . The main contribution of this work is as follows:(a) We formulate the multi-bit quantization as an optimization problem . The binary codes {−1, +1} are learned instead of rule-based. For the first time, we observe that the codes can be optimally derived by the binary search tree once the coefficients are knowns in advance, see, e.g., Algorithm 1. Thus the whole optimization is eased by removing the discrete unknowns , which are very difficult to handle. (b) We propose to use alternating minimization to tackle the quantization problem. By separating the binary codes and real coefficients into two parts, we can solve the subproblem efficiently when one part is fixed. With proper initialization, we only need two alternating cycles to get high precision approximation, which is effective enough to even quantize the activations on-line. (c) We systematically evaluate the effectiveness of our alternating quantization on language models.Two well-known RNN structures, i.e., LSTM and GRU, are tested with different quantization bits. Compared with the full-precision counterpart, by 2-bit quantization we can achieve ∼16× memory saving and ∼6× real inference acceleration on CPUs, with a reasonable loss on the accuracy. By 3-bit quantization, we can achieve almost no loss in accuracy or even surpass the original model with ∼10.5× memory saving and ∼3× real inference acceleration. Both results beat the exiting quantization works with large margins. To illustrate that our alternating quantization is very general to extend, we apply it to image classification tasks. In both RNNs and feedforward neural networks, the technique still achieves very plausible performance. In this work, we address the limitations of RNNs, i.e., large memory and high latency, by quantization. We formulate the quantization by minimizing the approximation error. Under the key observation that some parameters can be singled out when others fixed, a simple yet effective alternating method is proposed. We apply it to quantize LSTM and GRU on language models. By 2-bit weights and activations, we achieve only a reasonably accuracy loss compared with full precision one, with ∼16× reduction in memory and ∼6× real acceleration on CPUs. By 3-bit quantization, we can attain compatible or even better result than the full precision one, with ∼10.5× reduction in memory and ∼3× real acceleration. Both beat existing works with a large margin. We also apply our alternating quantization to image classification tasks. In both RNNs and feedforward neural networks, the method can still achieve very plausible performance. In this section, we discuss the implementation of the binary multiplication kernel in CPUs. The binary multiplication is divided into two steps: Entry-wise XNOR operation (corresponding to entry-wise product in the full precision multiplication) and bit count operation for accumulation (corresponding to compute the sum of all multiplied entries in the full precision multiplication). We test it on Intel Xeon E5-2682 v4 @ 2.50 GHz CPU. For the XNOR operation, we use the Single instruction, multiple data (SIMD) _mm256 _xor _ps, which can execute 256 bit simultaneously. For the bit count operation, we use the function _popcnt64 (Note that this step can further be accelerated by the up-coming instruction _mm512 _popcnt_epi64 , which can execute 512 bits simultaneously. Similarly, the XNOR operation can also be further accelerated by the up-coming _mm512 _xor _ps instruction to execute 512 bits simultaneously). We compare with the much optimized Intel Math Kernel Library (MKL) on full precision matrix vector multiplication and execute all codes in the single-thread mode. We conduct two scales of experiments: a matrix of size 4096 × 1024 multiplying a vector of size 1024 and a matrix of size 42000 × 1024 multiplying a vector of size 1024, which respectively correspond to the hidden state product W h h t−1 and the softmax layer W s h t for Text8 dataset during inference with batch size of 1 (See Eq. (6)). The results are shown in TAB6 . We can see that our alternating quantization step only accounts for a small portion of the total executing time, especially for the larger scale matrix vector multiplication. Compared with the full precision one, the binary multiplication can roughly achieve 6× acceleration with 2-bit quantization and 3× acceleration with 3-bit quantization. Note that this is only a simple test on CPU. Our alternating quantization method can also be extended to GPU, ASIC, and FPGA.
We propose a  new  quantization method and apply it to quantize RNNs for both compression and acceleration This paper proposes a multi-bit quantization method for recurrent neural networks. A technique for quantizing neural network weight matrices, and an alternating optimization procedure to estimate the set of k binary vectors and coefficients that best represent the original vector.