Which mode is better for federated learning? Centralized or
  Decentralized
Yan Sun Li Shen Dacheng Tao
Both centralized and decentralized approaches have shown excellent
performance and great application value in federated learning (FL). However,
current studies do not provide sufficient evidence to show which one performs
better. Although from the optimization perspective, decentralized methods can
approach the comparable convergence of centralized methods with less
communication, its test performance has always been inefficient in empirical
studies. To comprehensively explore their behaviors in FL, we study their
excess risks, including the joint analysis of both optimization and
generalization. We prove that on smooth non-convex objectives, 1) centralized
FL (CFL) always generalizes better than decentralized FL (DFL); 2) from
perspectives of the excess risk and test error in CFL, adopting partial
participation is superior to full participation; and, 3) there is a necessary
requirement for the topology in DFL to avoid performance collapse as the
training scale increases. Based on some simple hardware metrics, we could
evaluate which framework is better in practice. Extensive experiments are
conducted on common setups in FL to validate that our theoretical analysis is
contextually valid in practical scenarios.