RandCom: Random Communication Skipping Method for Decentralized
  Stochastic Optimization
Luyao Guo Sulaiman A. Alghunaim Kun Yuan Laurent Condat Jinde Cao
Distributed optimization methods with random communication skips are gaining
increasing attention due to their proven benefits in accelerating communication
complexity. Nevertheless, existing research mainly focuses on centralized
communication protocols for strongly convex deterministic settings. In this
work, we provide a decentralized optimization method called RandCom, which
incorporates probabilistic local updates. We analyze the performance of RandCom
in stochastic non-convex, convex, and strongly convex settings and demonstrate
its ability to asymptotically reduce communication overhead by the probability
of communication. Additionally, we prove that RandCom achieves linear speedup
as the number of nodes increases. In stochastic strongly convex settings, we
further prove that RandCom can achieve linear speedup with network-independent
stepsizes. Moreover, we apply RandCom to federated learning and provide
positive results concerning the potential for achieving linear speedup and the
suitability of the probabilistic local update approach for non-convex settings.