FedSplitX: Federated Split Learning for Computationally-Constrained
  Heterogeneous Clients
Jiyun Shin Jinhyun Ahn Honggu Kang Joonhyuk Kang
Foundation models (FMs) have demonstrated remarkable performance in machine
learning but demand extensive training data and computational resources.
Federated learning (FL) addresses the challenges posed by FMs, especially
related to data privacy and computational burdens. However, FL on FMs faces
challenges in situations with heterogeneous clients possessing varying
computing capabilities, as clients with limited capabilities may struggle to
train the computationally intensive FMs. To address these challenges, we
propose FedSplitX, a novel FL framework that tackles system heterogeneity.
FedSplitX splits a large model into client-side and server-side components at
multiple partition points to accommodate diverse client capabilities. This
approach enables clients to collaborate while leveraging the server's
computational power, leading to improved model performance compared to
baselines that limit model size to meet the requirement of the poorest client.
Furthermore, FedSplitX incorporates auxiliary networks at each partition point
to reduce communication costs and delays while enhancing model performance. Our
experiments demonstrate that FedSplitX effectively utilizes server capabilities
to train large models, outperforming baseline approaches.