Efficient Federated Learning via Local Adaptive Amended Optimizer with
  Linear Speedup
Yan Sun Li Shen Hao Sun Liang Ding Dacheng Tao
Adaptive optimization has achieved notable success for distributed learning
while extending adaptive optimizer to federated Learning (FL) suffers from
severe inefficiency, including (i) rugged convergence due to inaccurate
gradient estimation in global adaptive optimizer; (ii) client drifts
exacerbated by local over-fitting with the local adaptive optimizer. In this
work, we propose a novel momentum-based algorithm via utilizing the global
gradient descent and locally adaptive amended optimizer to tackle these
difficulties. Specifically, we incorporate a locally amended technique to the
adaptive optimizer, named Federated Local ADaptive Amended optimizer
(\textit{FedLADA}), which estimates the global average offset in the previous
communication round and corrects the local offset through a momentum-like term
to further improve the empirical training speed and mitigate the heterogeneous
over-fitting. Theoretically, we establish the convergence rate of
\textit{FedLADA} with a linear speedup property on the non-convex case under
the partial participation settings. Moreover, we conduct extensive experiments
on the real-world dataset to demonstrate the efficacy of our proposed
\textit{FedLADA}, which could greatly reduce the communication rounds and
achieves higher accuracy than several baselines.