FAGH: Accelerating Federated Learning with Approximated Global Hessian
Mrinmay Sen A. K. Qin Krishna Mohan C
In federated learning (FL), the significant communication overhead due to the
slow convergence speed of training the global model poses a great challenge.
Specifically, a large number of communication rounds are required to achieve
the convergence in FL. One potential solution is to employ the Newton-based
optimization method for training, known for its quadratic convergence rate.
However, the existing Newton-based FL training methods suffer from either
memory inefficiency or high computational costs for local clients or the
server. To address this issue, we propose an FL with approximated global
Hessian (FAGH) method to accelerate FL training. FAGH leverages the first
moment of the approximated global Hessian and the first moment of the global
gradient to train the global model. By harnessing the approximated global
Hessian curvature, FAGH accelerates the convergence of global model training,
leading to the reduced number of communication rounds and thus the shortened
training time. Experimental results verify FAGH's effectiveness in decreasing
the number of communication rounds and the time required to achieve the
pre-specified objectives of the global model performance in terms of training
and test losses as well as test accuracy. Notably, FAGH outperforms several
state-of-the-art FL training methods.