Byzantine-Robust Decentralized Stochastic Optimization with Stochastic
  Gradient Noise-Independent Learning Error
Jie Peng Weiyu Li Qing Ling
This paper studies Byzantine-robust stochastic optimization over a
decentralized network, where every agent periodically communicates with its
neighbors to exchange local models, and then updates its own local model by
stochastic gradient descent (SGD). The performance of such a method is affected
by an unknown number of Byzantine agents, which conduct adversarially during
the optimization process. To the best of our knowledge, there is no existing
work that simultaneously achieves a linear convergence speed and a small
learning error. We observe that the learning error is largely dependent on the
intrinsic stochastic gradient noise. Motivated by this observation, we
introduce two variance reduction methods, stochastic average gradient algorithm
(SAGA) and loopless stochastic variance-reduced gradient (LSVRG), to
Byzantine-robust decentralized stochastic optimization for eliminating the
negative effect of the stochastic gradient noise. The two resulting methods,
BRAVO-SAGA and BRAVO-LSVRG, enjoy both linear convergence speeds and stochastic
gradient noise-independent learning errors. Such learning errors are optimal
for a class of methods based on total variation (TV)-norm regularization and
stochastic subgradient update. We conduct extensive numerical experiments to
demonstrate their effectiveness under various Byzantine attacks.