Improving Group Connectivity for Generalization of Federated Deep
  Learning
Zexi Li Jie Lin Zhiqi Li Didi Zhu Chao Wu
Federated learning (FL) involves multiple heterogeneous clients
collaboratively training a global model via iterative local updates and model
fusion. The generalization of FL's global model has a large gap compared with
centralized training, which is its bottleneck for broader applications. In this
paper, we study and improve FL's generalization through a fundamental
``connectivity'' perspective, which means how the local models are connected in
the parameter region and fused into a generalized global model. The term
``connectivity'' is derived from linear mode connectivity (LMC), studying the
interpolated loss landscape of two different solutions (e.g., modes) of neural
networks. Bridging the gap between LMC and FL, in this paper, we leverage fixed
anchor models to empirically and theoretically study the transitivity property
of connectivity from two models (LMC) to a group of models (model fusion in
FL). Based on the findings, we propose FedGuCci and FedGuCci+, improving group
connectivity for better generalization. It is shown that our methods can boost
the generalization of FL under client heterogeneity across various tasks (4 CV
datasets and 6 NLP datasets), models (both convolutional and
transformer-based), and training paradigms (both from-scratch and
pretrain-finetune).