FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup
  for Non-IID Data
Hao Sun Li Shen Shixiang Chen Jingwei Sun Jing Li Guangzhong Sun Dacheng Tao
Federated learning is an emerging distributed machine learning method,
enables a large number of clients to train a model without exchanging their
local data. The time cost of communication is an essential bottleneck in
federated learning, especially for training large-scale deep neural networks.
Some communication-efficient federated learning methods, such as FedAvg and
FedAdam, share the same learning rate across different clients. But they are
not efficient when data is heterogeneous. To maximize the performance of
optimization methods, the main challenge is how to adjust the learning rate
without hurting the convergence. In this paper, we propose a heterogeneous
local variant of AMSGrad, named FedLALR, in which each client adjusts its
learning rate based on local historical gradient squares and synchronized
learning rates. Theoretical analysis shows that our client-specified auto-tuned
learning rate scheduling can converge and achieve linear speedup with respect
to the number of clients, which enables promising scalability in federated
optimization. We also empirically compare our method with several
communication-efficient federated optimization methods. Extensive experimental
results on Computer Vision (CV) tasks and Natural Language Processing (NLP)
task show the efficacy of our proposed FedLALR method and also coincides with
our theoretical findings.