Data-parallel neural network training is network-intensive, so gradient dropping was designed to exchange only large gradients.   However, gradient dropping has been shown to slow convergence.   We propose to improve convergence by having each node combine its locally computed gradient with the sparse global gradient exchanged over the network. We empirically confirm with machine translation tasks that gradient dropping with local gradients approaches convergence 48% faster than non-compressed multi-node training and 28% faster compared to vanilla gradient dropping. We also show that gradient dropping with a local gradient update does not reduce the model's final quality. Training a neural network can be slow, especially with a large model or dataset BID12 BID18 . Distributed training is becoming essential to speed up the process. In data-parallel training, multiple workers optimize the same parameters based on different parts of the training data then exchange parameters.Data-parallel training is network intensive because workers send and fetch gradients that have the same size as the model. Several techniques have been proposed to reduce the traffic in dataparallelism training by using quantization to compress the gradient sent BID13 BID1 or selecting sparse matrices BID17 BID5 BID0 BID10 .Gradient dropping, and its extension Deep Gradient Compression BID10 , is a recent approach that compresses the network by sending a small fraction (about 1%) of the largest gradients (by absolute value). This technique is based on the observation that the gradient values are skewed, as most are close to zero. An issue with gradient compression is that gradients are compressed so much that it slows the model's convergence rate and can reduce the model's final quality BID0 .In vanilla gradient dropping, all nodes update with the same sparse gradient exchanged over the network, while other parameters are unchanged. However, each node has computed a local gradient on its own data. Can we exploit this dense local gradient alongside the sparse global gradient to improve convergence? We propose and evaluate three ways to combine them. We significantly reduce convergence damage caused by compressing the gradient through gradient dropping in data-parallelism training. We utilize a locally-computed gradient to predict and reconstruct the dense gradient. Our experiments show that we can improve the training time up to 45% faster compared to a non-compressed multi-node system and 3x faster compared to a single-node system. Local gradient update is also empirically shown to negate the quality loss caused by gradient dropping.
We improve gradient dropping (a technique of only exchanging large gradients on distributed training) by incorporating local gradients while doing a parameter update to reduce quality loss and further improve the training time. This paper proposes a 3 modes for combining local and global gradients to better use more computing nodes Looks at the problem of reducing the communication requirement for implementing the distributed optimiztion techniques, particularly SGD