Leveraging Foundation Models to Improve Lightweight Clients in Federated
  Learning
Xidong Wu Wan-Yi Lin Devin Willmott Filipe Condessa Yufei Huang Zhenzhen Li Madan Ravi Ganesh
Federated Learning (FL) is a distributed training paradigm that enables
clients scattered across the world to cooperatively learn a global model
without divulging confidential data. However, FL faces a significant challenge
in the form of heterogeneous data distributions among clients, which leads to a
reduction in performance and robustness. A recent approach to mitigating the
impact of heterogeneous data distributions is through the use of foundation
models, which offer better performance at the cost of larger computational
overheads and slower inference speeds. We introduce foundation model
distillation to assist in the federated training of lightweight client models
and increase their performance under heterogeneous data settings while keeping
inference costs low. Our results show improvement in the global model
performance on a balanced testing set, which contains rarely observed samples,
even under extreme non-IID client data distributions. We conduct a thorough
evaluation of our framework with different foundation model backbones on
CIFAR10, with varying degrees of heterogeneous data distributions ranging from
class-specific data partitions across clients to dirichlet data sampling,
parameterized by values between 0.01 and 1.0.