Distributed Momentum Methods Under Biased Gradient Estimations
Ali Beikmohammadi Sarit Khirirat Sindri Magnússon
Distributed stochastic gradient methods are gaining prominence in solving
large-scale machine learning problems that involve data distributed across
multiple nodes. However, obtaining unbiased stochastic gradients, which have
been the focus of most theoretical research, is challenging in many distributed
machine learning applications. The gradient estimations easily become biased,
for example, when gradients are compressed or clipped, when data is shuffled,
and in meta-learning and reinforcement learning. In this work, we establish
non-asymptotic convergence bounds on distributed momentum methods under biased
gradient estimation on both general non-convex and $\mu$-PL non-convex
problems. Our analysis covers general distributed optimization problems, and we
work out the implications for special cases where gradient estimates are
biased, i.e., in meta-learning and when the gradients are compressed or
clipped. Our numerical experiments on training deep neural networks with
Top-$K$ sparsification and clipping verify faster convergence performance of
momentum methods than traditional biased gradient descent.