Distributed Learning of Mixtures of Experts
Faïcel Chamroukhi Nhat Thien Pham
In modern machine learning problems we deal with datasets that are either
distributed by nature or potentially large for which distributing the
computations is usually a standard way to proceed, since centralized algorithms
are in general ineffective. We propose a distributed learning approach for
mixtures of experts (MoE) models with an aggregation strategy to construct a
reduction estimator from local estimators fitted parallelly to distributed
subsets of the data. The aggregation is based on an optimal minimization of an
expected transportation divergence between the large MoE composed of local
estimators and the unknown desired MoE model. We show that the provided
reduction estimator is consistent as soon as the local estimators to be
aggregated are consistent, and its construction is performed by a proposed
majorization-minimization (MM) algorithm that is computationally effective. We
study the statistical and numerical properties for the proposed reduction
estimator on experiments that demonstrate its performance compared to namely
the global estimator constructed in a centralized way from the full dataset.
For some situations, the computation time is more than ten times faster, for a
comparable performance. Our source codes are publicly available on Github.