SignSGD with Federated Defense: Harnessing Adversarial Attacks through
  Gradient Sign Decoding
Chanho Park Namyoon Lee
Distributed learning is an effective approach to accelerate model training
using multiple workers. However, substantial communication delays emerge
between workers and a parameter server due to massive costs associated with
communicating gradients. SignSGD with majority voting (signSGD-MV) is a simple
yet effective optimizer that reduces communication costs through one-bit
quantization, yet the convergence rates considerably decrease as adversarial
workers increase. In this paper, we show that the convergence rate is invariant
as the number of adversarial workers increases, provided that the number of
adversarial workers is smaller than that of benign workers. The key idea
showing this counter-intuitive result is our novel signSGD with federated
defense (signSGD-FD). Unlike the traditional approaches, signSGD-FD exploits
the gradient information sent by adversarial workers with the proper weights,
which are obtained through gradient sign decoding. Experimental results
demonstrate signSGD-FD achieves superior convergence rates over traditional
algorithms in various adversarial attack scenarios.