FedBIAD: Communication-Efficient and Accuracy-Guaranteed Federated
  Learning with Bayesian Inference-Based Adaptive Dropout
Jingjing Xue Min Liu Sheng Sun Yuwei Wang Hui Jiang Xuefeng Jiang
Federated Learning (FL) emerges as a distributed machine learning paradigm
without end-user data transmission, effectively avoiding privacy leakage.
Participating devices in FL are usually bandwidth-constrained, and the uplink
is much slower than the downlink in wireless networks, which causes a severe
uplink communication bottleneck. A prominent direction to alleviate this
problem is federated dropout, which drops fractional weights of local models.
However, existing federated dropout studies focus on random or ordered dropout
and lack theoretical support, resulting in unguaranteed performance. In this
paper, we propose Federated learning with Bayesian Inference-based Adaptive
Dropout (FedBIAD), which regards weight rows of local models as probability
distributions and adaptively drops partial weight rows based on importance
indicators correlated with the trend of local training loss. By applying
FedBIAD, each client adaptively selects a high-quality dropping pattern with
accurate approximations and only transmits parameters of non-dropped weight
rows to mitigate uplink costs while improving accuracy. Theoretical analysis
demonstrates that the convergence rate of the average generalization error of
FedBIAD is minimax optimal up to a squared logarithmic factor. Extensive
experiments on image classification and next-word prediction show that compared
with status quo approaches, FedBIAD provides 2x uplink reduction with an
accuracy increase of up to 2.41% even on non-Independent and Identically
Distributed (non-IID) data, which brings up to 72% decrease in training time.