Communication-Efficient Heterogeneous Federated Learning with
  Generalized Heavy-Ball Momentum
Riccardo Zaccone Carlo Masone Marco Ciccone
Federated Learning (FL) is the state-of-the-art approach for learning from
decentralized data in privacy-constrained scenarios. As the current literature
reports, the main problems associated with FL refer to system and statistical
challenges: the former ones demand for efficient learning from edge devices,
including lowering communication bandwidth and frequency, while the latter
require algorithms robust to non-iidness. State-of-art approaches either
guarantee convergence at increased communication cost or are not sufficiently
robust to handle extreme heterogeneous local distributions. In this work we
propose a novel generalization of the heavy-ball momentum, and present FedHBM
to effectively address statistical heterogeneity in FL without introducing any
communication overhead. We conduct extensive experimentation on common FL
vision and NLP datasets, showing that our FedHBM algorithm empirically yields
better model quality and higher convergence speed w.r.t. the state-of-art,
especially in pathological non-iid scenarios. While being designed for
cross-silo settings, we show how FedHBM is applicable in moderate-to-high
cross-device scenarios, and how good model initializations (e.g. pre-training)
can be exploited for prompt acceleration. Extended experimentation on
large-scale real-world federated datasets further corroborates the
effectiveness of our approach for real-world FL applications.