FedDec: Peer-to-peer Aided Federated Learning
Marina Costantini Giovanni Neglia Thrasyvoulos Spyropoulos
Federated learning (FL) has enabled training machine learning models
exploiting the data of multiple agents without compromising privacy. However,
FL is known to be vulnerable to data heterogeneity, partial device
participation, and infrequent communication with the server, which are
nonetheless three distinctive characteristics of this framework. While much of
the recent literature has tackled these weaknesses using different tools, only
a few works have explored the possibility of exploiting inter-agent
communication to improve FL's performance. In this work, we present FedDec, an
algorithm that interleaves peer-to-peer communication and parameter averaging
(similar to decentralized learning in networks) between the local gradient
updates of FL. We analyze the convergence of FedDec under the assumptions of
non-iid data distribution, partial device participation, and smooth and
strongly convex costs, and show that inter-agent communication alleviates the
negative impact of infrequent communication rounds with the server by reducing
the dependence on the number of local updates $H$ from $O(H^2)$ to $O(H)$.
Furthermore, our analysis reveals that the term improved in the bound is
multiplied by a constant that depends on the spectrum of the inter-agent
communication graph, and that vanishes quickly the more connected the network
is. We confirm the predictions of our theory in numerical simulations, where we
show that FedDec converges faster than FedAvg, and that the gains are greater
as either $H$ or the connectivity of the network increase.