FedFisher: Leveraging Fisher Information for One-Shot Federated Learning
Divyansh Jhunjhunwala Shiqiang Wang Gauri Joshi
Standard federated learning (FL) algorithms typically require multiple rounds
of communication between the server and the clients, which has several
drawbacks, including requiring constant network connectivity, repeated
investment of computational resources, and susceptibility to privacy attacks.
One-Shot FL is a new paradigm that aims to address this challenge by enabling
the server to train a global model in a single round of communication. In this
work, we present FedFisher, a novel algorithm for one-shot FL that makes use of
Fisher information matrices computed on local client models, motivated by a
Bayesian perspective of FL. First, we theoretically analyze FedFisher for
two-layer over-parameterized ReLU neural networks and show that the error of
our one-shot FedFisher global model becomes vanishingly small as the width of
the neural networks and amount of local training at clients increases. Next, we
propose practical variants of FedFisher using the diagonal Fisher and K-FAC
approximation for the full Fisher and highlight their communication and compute
efficiency for FL. Finally, we conduct extensive experiments on various
datasets, which show that these variants of FedFisher consistently improve over
competing baselines.