Securing Distributed SGD against Gradient Leakage Threats
Wenqi Wei Ling Liu Jingya Zhou Ka-Ho Chow Yanzhao Wu
This paper presents a holistic approach to gradient leakage resilient
distributed Stochastic Gradient Descent (SGD). First, we analyze two types of
strategies for privacy-enhanced federated learning: (i) gradient pruning with
random selection or low-rank filtering and (ii) gradient perturbation with
additive random noise or differential privacy noise. We analyze the inherent
limitations of these approaches and their underlying impact on privacy
guarantee, model accuracy, and attack resilience. Next, we present a gradient
leakage resilient approach to securing distributed SGD in federated learning,
with differential privacy controlled noise as the tool. Unlike conventional
methods with the per-client federated noise injection and fixed noise parameter
strategy, our approach keeps track of the trend of per-example gradient
updates. It makes adaptive noise injection closely aligned throughout the
federated model training. Finally, we provide an empirical privacy analysis on
the privacy guarantee, model utility, and attack resilience of the proposed
approach. Extensive evaluation using five benchmark datasets demonstrates that
our gradient leakage resilient approach can outperform the state-of-the-art
methods with competitive accuracy performance, strong differential privacy
guarantee, and high resilience against gradient leakage attacks. The code
associated with this paper can be found:
https://github.com/git-disl/Fed-alphaCDP.