Federated Learning with Nonvacuous Generalisation Bounds
Pierre Jobic Maxime Haddouche Benjamin Guedj
We introduce a novel strategy to train randomised predictors in federated
learning, where each node of the network aims at preserving its privacy by
releasing a local predictor but keeping secret its training dataset with
respect to the other nodes. We then build a global randomised predictor which
inherits the properties of the local private predictors in the sense of a
PAC-Bayesian generalisation bound. We consider the synchronous case where all
nodes share the same training objective (derived from a generalisation bound),
and the asynchronous case where each node may have its own personalised
training objective. We show through a series of numerical experiments that our
approach achieves a comparable predictive performance to that of the batch
approach where all datasets are shared across nodes. Moreover the predictors
are supported by numerically nonvacuous generalisation bounds while preserving
privacy for each node. We explicitly compute the increment on predictive
performance and generalisation bounds between batch and federated settings,
highlighting the price to pay to preserve privacy.