A Masked Pruning Approach for Dimensionality Reduction in
  Communication-Efficient Federated Learning Systems
Tamir L. S. Gez Kobi Cohen
Federated Learning (FL) represents a growing machine learning (ML) paradigm
designed for training models across numerous nodes that retain local datasets,
all without directly exchanging the underlying private data with the parameter
server (PS). Its increasing popularity is attributed to notable advantages in
terms of training deep neural network (DNN) models under privacy aspects and
efficient utilization of communication resources. Unfortunately, DNNs suffer
from high computational and communication costs, as well as memory consumption
in intricate tasks. These factors restrict the applicability of FL algorithms
in communication-constrained systems with limited hardware resources.
  In this paper, we develop a novel algorithm that overcomes these limitations
by synergistically combining a pruning-based method with the FL process,
resulting in low-dimensional representations of the model with minimal
communication cost, dubbed Masked Pruning over FL (MPFL). The algorithm
operates by initially distributing weights to the nodes through the PS.
Subsequently, each node locally trains its model and computes pruning masks.
These low-dimensional masks are then transmitted back to the PS, which
generates a consensus pruning mask, broadcasted back to the nodes. This
iterative process enhances the robustness and stability of the masked pruning
model. The generated mask is used to train the FL model, achieving significant
bandwidth savings. We present an extensive experimental study demonstrating the
superior performance of MPFL compared to existing methods. Additionally, we
have developed an open-source software package for the benefit of researchers
and developers in related fields.