FedFa: A Fully Asynchronous Training Paradigm for Federated Learning
Haotian Xu Zhaorui Zhang Sheng Di Benben Liu Alharthi Khalid Jiannong Cao
Federated learning has been identified as an efficient decentralized training
paradigm for scaling the machine learning model training on a large number of
devices while guaranteeing the data privacy of the trainers. FedAvg has become
a foundational parameter update strategy for federated learning, which has been
promising to eliminate the effect of the heterogeneous data across clients and
guarantee convergence. However, the synchronization parameter update barriers
for each communication round during the training significant time on waiting,
slowing down the training procedure. Therefore, recent state-of-the-art
solutions propose using semi-asynchronous approaches to mitigate the waiting
time cost with guaranteed convergence. Nevertheless, emerging semi-asynchronous
approaches are unable to eliminate the waiting time completely.
  We propose a full asynchronous training paradigm, called FedFa, which can
guarantee model convergence and eliminate the waiting time completely for
federated learning by using a few buffered results on the server for parameter
updating. Further, we provide theoretical proof of the convergence rate for our
proposed FedFa. Extensive experimental results indicate our approach
effectively improves the training performance of federated learning by up to 6x
and 4x speedup compared to the state-of-the-art synchronous and
semi-asynchronous strategies while retaining high accuracy in both IID and
Non-IID scenarios.