How to Collaborate: Towards Maximizing the Generalization Performance in
  Cross-Silo Federated Learning
Yuchang Sun Marios Kountouris Jun Zhang
Federated learning (FL) has attracted vivid attention as a privacy-preserving
distributed learning framework. In this work, we focus on cross-silo FL, where
clients become the model owners after training and are only concerned about the
model's generalization performance on their local data. Due to the data
heterogeneity issue, asking all the clients to join a single FL training
process may result in model performance degradation. To investigate the
effectiveness of collaboration, we first derive a generalization bound for each
client when collaborating with others or when training independently. We show
that the generalization performance of a client can be improved only by
collaborating with other clients that have more training data and similar data
distribution. Our analysis allows us to formulate a client utility maximization
problem by partitioning clients into multiple collaborating groups. A
hierarchical clustering-based collaborative training (HCCT) scheme is then
proposed, which does not need to fix in advance the number of groups. We
further analyze the convergence of HCCT for general non-convex loss functions
which unveils the effect of data similarity among clients. Extensive
simulations show that HCCT achieves better generalization performance than
baseline schemes, whereas it degenerates to independent training and
conventional FL in specific scenarios.