Deep neural networks are usually huge, which significantly limits the deployment on low-end devices. In recent years, many
 weight-quantized models have  been proposed. They have small storage and fast inference, but training can still be time-consuming. This can be improved with distributed learning. To reduce the high communication cost due to worker-server synchronization, recently gradient quantization has also been proposed to train deep networks with full-precision weights. 
 In this paper, we theoretically study how the combination of both weight and gradient quantization affects convergence.
 We show  that (i) weight-quantized models converge to an error related to the weight quantization resolution and weight dimension; (ii) quantizing gradients slows convergence by a factor related to the gradient quantization resolution and dimension; and (iii) clipping the gradient before quantization renders this factor dimension-free, thus allowing the use of fewer bits for gradient quantization. Empirical experiments confirm the theoretical convergence results, and demonstrate that quantized networks can speed up training and have comparable performance as full-precision networks. Deep neural networks are usually huge. The high demand in time and space can significantly limit deployment on low-end devices. To alleviate this problem, many approaches have been recently proposed to compress deep networks. One direction is network quantization, which represents each network weight with a small number of bits. Besides significantly reducing the model size, it also accelerates network training and inference. Many weight quantization methods aim at approximating the full-precision weights in each iteration BID3 BID20 BID23 BID15 BID19 BID8 . Recently, loss-aware quantization minimizes the loss directly w.r.t. the quantized weights BID11 BID10 BID14 , and often achieves better performance than approximation-based methods.Distributed learning can further speed up training of weight-quantized networks BID6 . A key challenge is on reducing the expensive communication cost incurred during synchronization of the gradients and model parameters BID17 . Recently, algorithms that sparsify BID0 BID28 or quantize the gradients BID26 BID29 BID2 have been proposed.In this paper, we consider quantization of both the weights and gradients in a distributed environment. Quantizing both weights and gradients has been explored in the DoReFa-Net BID32 , QNN BID12 , WAGE BID30 and ZipML . We differ from them in two aspects. First, existing methods mainly consider learning on a single machine, and gradient quantization is used to reduce the computations in backpropagation. On the other hand, we consider a distributed environment, and use gradient quantization to reduce communication cost and accelerate distributed learning of weight-quantized networks. Second, while DoReFa-Net, QNN and WAGE show impressive empirical results on the quantized network, theoretical guarantees are not provided. ZipML provides convergence analysis, but is limited to stochastic weight quantization, square loss with the linear model, and requires the stochastic gradients to be unbiased. This can be restrictive as most state-of-the-art weight quantization methods BID23 BID20 BID15 BID8 BID11 BID10 ) are deterministic, and the resultant stochastic gradients are biased.In this paper, we relax the restrictions on the loss function, and study in an online learning setting how the gradient precision affects convergence of weight-quantized networks in a distributed environment. The main findings are:1. With either full-precision or quantized gradients, the average regret of loss-aware weight quantization does not converge to zero, but to an error related to the weight quantization resolution ∆ w and dimension d. The smaller the ∆ w or d, the smaller is the error (Theorems 1 and 2). 2. With either full-precision or quantized gradients, the average regret converges with a O(1/ √ T ) rate to the error, where T is the number of iterations. However, gradient quantization slows convergence (relative to using full-precision gradients) by a factor related to gradient quantization resolution ∆ g and d. The larger the ∆ g or d, the slower is the convergence FIG0 ). This can be problematic when (i) the weight quantized model has a large d (e.g., deep networks); and (ii) the communication cost is a bottleneck in the distributed setting, which favors a small number of bits for the gradients, and thus a large ∆ g . 3. For gradients following the normal distribution, gradient clipping renders the speed degradation mentioned above dimension-free. However, an additional error is incurred. The convergence speedup and error are related to how aggressive clipping is performed. More aggressive clipping results in faster convergence, but a larger error (Theorem 3). 4. Empirical results show that quantizing gradients significantly reduce communication cost, and gradient clipping makes speed degradation caused by gradient quantization negligible. With quantized clipped gradients, distributed training of weight-quantized networks is much faster, while comparable accuracy with the use of full-precision gradients is maintained (Section 4).Notations . For a vector x, √ x is the element-wise square root, x 2 is the element-wise square, Diag(x) returns a diagonal matrix with x on the diagonal, and x y is the element-wise multiplication of vectors x and y. For a matrix Q, x 2 Q = x Qx. For a matrix X, √ X is the element-wise square root, and diag(X) returns a vector extracted from the diagonal elements of X. In this paper, we studied loss-aware weight-quantized networks with quantized gradient for efficient communication in a distributed environment. Convergence analysis is provided for weight-quantized models with full-precision, quantized and quantized clipped gradients. Empirical experiments confirm the theoretical results, and demonstrate that quantized networks can speed up training and have comparable performance as full-precision networks.
In this paper, we studied efficient training of loss-aware weight-quantized  networks with  quantized gradient  in a distributed environment, both theoretically and empirically. This paper studies convergence properties of loss-aware weight quantization with different gradient precisions in the distributed environment, and provides convergence analysis for weight quantization with full-precision, quantized and quantized clipped gradients. The authors proposes an analysis of the effect of simultaneously quantizing the weights and gradients in training a parametrized model in a fully-synchronized distributed environment.