Federated Learning with Manifold Regularization and Normalized Update
  Reaggregation
Xuming An Li Shen Han Hu Yong Luo
Federated Learning (FL) is an emerging collaborative machine learning
framework where multiple clients train the global model without sharing their
own datasets. In FL, the model inconsistency caused by the local data
heterogeneity across clients results in the near-orthogonality of client
updates, which leads to the global update norm reduction and slows down the
convergence. Most previous works focus on eliminating the difference of
parameters (or gradients) between the local and global models, which may fail
to reflect the model inconsistency due to the complex structure of the machine
learning model and the Euclidean space's limitation in meaningful geometric
representations. In this paper, we propose FedMRUR by adopting the manifold
model fusion scheme and a new global optimizer to alleviate the negative
impacts. Concretely, FedMRUR adopts a hyperbolic graph manifold regularizer
enforcing the representations of the data in the local and global models are
close to each other in a low-dimensional subspace. Because the machine learning
model has the graph structure, the distance in hyperbolic space can reflect the
model bias better than the Euclidean distance. In this way, FedMRUR exploits
the manifold structures of the representations to significantly reduce the
model inconsistency. FedMRUR also aggregates the client updates norms as the
global update norm, which can appropriately enlarge each client's contribution
to the global update, thereby mitigating the norm reduction introduced by the
near-orthogonality of client updates. Furthermore, we theoretically prove that
our algorithm can achieve a linear speedup property for non-convex setting
under partial client participation.Experiments demonstrate that FedMRUR can
achieve a new state-of-the-art (SOTA) accuracy with less communication.