As an emerging field, federated learning has recently attracted considerable attention.
 Compared to distributed learning in the datacenter setting, federated learning
 has more strict constraints on computate efficiency of the learned model and communication
 cost during the training process. In this work, we propose an efficient
 federated learning framework based on variational dropout. Our approach is able
 to jointly learn a sparse model while reducing the amount of gradients exchanged
 during the iterative training process. We demonstrate the superior performance
 of our approach on achieving significant model compression and communication
 reduction ratios with no accuracy loss. Federated Learning is an emerging machine learning approach that has recently attracted considerable attention due to its wide range of applications in mobile scenarios BID18 BID12 BID24 . It enables geographically distributed devices such as mobile phones to collaboratively learn a shared model while keeping the training data on each phone. This is different from standard machine learning approach which requires all the training data to be centralized in a server or in a datacenter. As such, federated learning enables distributing the knowledge across phones without sharing users' private data.Federated Learning uses some form of distributed stochastic gradient descent (SGD) and requires a parameter server to coordinate the training process. The server initializes the model and distributes it to all the participating devices. In each distributed SGD iteration, each device computes the gradients of the model parameters using its local data. The server aggregates the gradients from each device, averages them, and sends the averaged gradients back. Each device then updates the model parameters using the averaged gradients. In such manner, each device benefits from obtaining a better model than the one trained only on the locally stored private data.While federated learning shares some common features with distributed learning in the datacenter setting BID3 BID16 since they both use distributed SGD as the core training technique, federated learning has two more strict constraints which datacenter setting does not have:Model Constraint: Compared to datacenters, mobile devices have much less compute resources. This requires the final model learned in the federated learning setting to be computationally efficient so that it can efficiently run on mobile devices.Communication Constraint: In datacenters, communication between the server and working nodes during SGD is conducted via Gbps Ethernet or InfiniBand network with even higher bandwidth BID26 . In contrast, communication in the federated learning setting relies on wireless networks such as 4G and Wi-Fi. Both uplink and downlink bandwidths of those wireless networks are at Mbps scale, which is much lower than the Gbps scale in the datacenter setting. The limited bandwidth in the federated learning setting illustrates the necessity of reducing the communication cost to accelerate the training process.In this work, we propose an efficient federated learning framework that meets both model and communication constraints. Our approach is inspired by variational dropout BID10 . Our key idea is to jointly and iteratively sparsify the parameters of the shared model to be learned as well as the gradients exchanged between the server and the participating devices during the distributed SGD training process. By sparsifying parameters, only important parameters are kept, and the final model learned thus becomes computationally efficient run on mobile devices. By sparsifying gradients, only important gradients are transmitted, and the communication cost is thus significantly reduced. We examine the performance of our framework on three deep neural networks and five datasets that fit the federated learning setting and are appropriate to be deployed on resource-limited mobile devices. Our experiment results show that our framework is able to achieve significant model compression and communication reduction ratios with no accuracy loss.
a joint model and gradient sparsification method for federated learning Applies variational dropout to reduce the communication cost of distributed training of neural networks, and does experiments on mnist, cifar10 and svhn datasets.  The authors propose an algorithm that reduces communication costs in federated learning by sending sparse gradients from device to server and back. Combines distributed optimization algorithm with variational dropout to sparsify the gradients sent to master server from local learners.