Training Heterogeneous Client Models using Knowledge Distillation in
  Serverless Federated Learning
Mohak Chadha Pulkit Khera Jianfeng Gu Osama Abboud Michael Gerndt
Federated Learning (FL) is an emerging machine learning paradigm that enables
the collaborative training of a shared global model across distributed clients
while keeping the data decentralized. Recent works on designing systems for
efficient FL have shown that utilizing serverless computing technologies,
particularly Function-as-a-Service (FaaS) for FL, can enhance resource
efficiency, reduce training costs, and alleviate the complex infrastructure
management burden on data holders. However, existing serverless FL systems
implicitly assume a uniform global model architecture across all participating
clients during training. This assumption fails to address fundamental
challenges in practical FL due to the resource and statistical data
heterogeneity among FL clients. To address these challenges and enable
heterogeneous client models in serverless FL, we utilize Knowledge Distillation
(KD) in this paper. Towards this, we propose novel optimized serverless
workflows for two popular conventional federated KD techniques, i.e., FedMD and
FedDF. We implement these workflows by introducing several extensions to an
open-source serverless FL system called FedLess. Moreover, we comprehensively
evaluate the two strategies on multiple datasets across varying levels of
client data heterogeneity using heterogeneous client models with respect to
accuracy, fine-grained training times, and costs. Results from our experiments
demonstrate that serverless FedDF is more robust to extreme non-IID data
distributions, is faster, and leads to lower costs than serverless FedMD. In
addition, compared to the original implementation, our optimizations for
particular steps in FedMD and FedDF lead to an average speedup of 3.5x and
1.76x across all datasets.