Distributed Collapsed Gibbs Sampler for Dirichlet Process Mixture Models
  in Federated Learning
Reda Khoufache Mustapha Lebbah Hanene Azzag Etienne Goffinet Djamel Bouchaffra
Dirichlet Process Mixture Models (DPMMs) are widely used to address
clustering problems. Their main advantage lies in their ability to
automatically estimate the number of clusters during the inference process
through the Bayesian non-parametric framework. However, the inference becomes
considerably slow as the dataset size increases. This paper proposes a new
distributed Markov Chain Monte Carlo (MCMC) inference method for DPMMs (DisCGS)
using sufficient statistics. Our approach uses the collapsed Gibbs sampler and
is specifically designed to work on distributed data across independent and
heterogeneous machines, which habilitates its use in horizontal federated
learning. Our method achieves highly promising results and notable scalability.
For instance, with a dataset of 100K data points, the centralized algorithm
requires approximately 12 hours to complete 100 iterations while our approach
achieves the same number of iterations in just 3 minutes, reducing the
execution time by a factor of 200 without compromising clustering performance.
The code source is publicly available at
https://github.com/redakhoufache/DisCGS.