Bayesian Federated Learning Via Expectation Maximization and Turbo Deep
  Approximate Message Passing
Wei Xu An Liu Yiting Zhang Vincent Lau
Federated learning (FL) is a machine learning paradigm where the clients
possess decentralized training data and the central server handles aggregation
and scheduling. Typically, FL algorithms involve clients training their local
models using stochastic gradient descent (SGD), which carries drawbacks such as
slow convergence and being prone to getting stuck in suboptimal solutions. In
this work, we propose a message passing based Bayesian federated learning (BFL)
framework to avoid these drawbacks.Specifically, we formulate the problem of
deep neural network (DNN) learning and compression and as a sparse Bayesian
inference problem, in which group sparse prior is employed to achieve
structured model compression. Then, we propose an efficient BFL algorithm
called EMTDAMP, where expectation maximization (EM) and turbo deep approximate
message passing (TDAMP) are combined to achieve distributed learning and
compression. The central server aggregates local posterior distributions to
update global posterior distributions and update hyperparameters based on EM to
accelerate convergence. The clients perform TDAMP to achieve efficient
approximate message passing over DNN with joint prior distribution. We detail
the application of EMTDAMP to Boston housing price prediction and handwriting
recognition, and present extensive numerical results to demonstrate the
advantages of EMTDAMP.