Asymmetrically Decentralized Federated Learning
Qinglun Li Miao Zhang Nan Yin Quanjun Yin Li Shen
To address the communication burden and privacy concerns associated with the
centralized server in Federated Learning (FL), Decentralized Federated Learning
(DFL) has emerged, which discards the server with a peer-to-peer (P2P)
communication framework. However, most existing DFL algorithms are based on
symmetric topologies, such as ring and grid topologies, which can easily lead
to deadlocks and are susceptible to the impact of network link quality in
practice. To address these issues, this paper proposes the DFedSGPSM algorithm,
which is based on asymmetric topologies and utilizes the Push-Sum protocol to
effectively solve consensus optimization problems. To further improve algorithm
performance and alleviate local heterogeneous overfitting in Federated Learning
(FL), our algorithm combines the Sharpness Aware Minimization (SAM) optimizer
and local momentum. The SAM optimizer employs gradient perturbations to
generate locally flat models and searches for models with uniformly low loss
values, mitigating local heterogeneous overfitting. The local momentum
accelerates the optimization process of the SAM optimizer. Theoretical analysis
proves that DFedSGPSM achieves a convergence rate of
$\mathcal{O}(\frac{1}{\sqrt{T}})$ in a non-convex smooth setting under mild
assumptions. This analysis also reveals that better topological connectivity
achieves tighter upper bounds. Empirically, extensive experiments are conducted
on the MNIST, CIFAR10, and CIFAR100 datasets, demonstrating the superior
performance of our algorithm compared to state-of-the-art optimizers.