FedMS: Federated Learning with Mixture of Sparsely Activated Foundations
  Models
Panlong Wu Kangshuo Li Ting Wang Fangxin Wang
Foundation models have shown great success in natural language processing,
computer vision, and multimodal tasks. FMs have a large number of model
parameters, thus requiring a substantial amount of data to help optimize the
model during the training. Federated learning has revolutionized machine
learning by enabling collaborative learning from decentralized data while still
preserving the data privacy of clients. Despite the great benefits foundation
models can have empowered by federated learning, they face severe computation,
communication, and statistical challenges. In this paper, we propose a novel
two-stage federated learning algorithm called FedMS. A global expert is trained
in the first stage and a local expert is trained in the second stage to provide
better personalization. We construct a Mixture of Foundation Models (MoFM) with
these two experts and design a gate neural network with an inserted gate
adapter that joins the aggregation every communication round in the second
stage. To further adapt to edge computing scenarios with limited computational
resources, we design a novel Sparsely Activated LoRA (SAL) algorithm that
freezes the pre-trained foundation model parameters inserts low-rank adaptation
matrices into transformer blocks and activates them progressively during the
training. We employ extensive experiments to verify the effectiveness of FedMS,
results show that FedMS outperforms other SOTA baselines by up to 55.25% in
default settings.