FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal
  Heterogeneous Federated Learning
Haokun Chen Yao Zhang Denis Krompass Jindong Gu Volker Tresp
Recently, foundation models have exhibited remarkable advancements in
multi-modal learning. These models, equipped with millions (or billions) of
parameters, typically require a substantial amount of data for finetuning.
However, collecting and centralizing training data from diverse sectors becomes
challenging due to distinct privacy regulations. Federated Learning (FL)
emerges as a promising solution, enabling multiple clients to collaboratively
train neural networks without centralizing their local data. To alleviate
client computation burdens and communication overheads, previous works have
adapted Parameter-efficient Finetuning (PEFT) methods for FL. Hereby, only a
small fraction of the model parameters are optimized and communicated during
federated communications. Nevertheless, most previous works have focused on a
single modality and neglected one common phenomenon, i.e., the presence of data
heterogeneity across the clients. Therefore, in this work, we propose a
finetuning framework tailored to heterogeneous multi-modal FL, called Federated
Dual-Aadapter Teacher (FedDAT). Specifically, our approach leverages a
Dual-Adapter Teacher (DAT) to address data heterogeneity by regularizing the
client local updates and applying Mutual Knowledge Distillation (MKD) for an
efficient knowledge transfer. FedDAT is the first approach that enables an
efficient distributed finetuning of foundation models for a variety of
heterogeneous Vision-Language tasks. To demonstrate its effectiveness, we
conduct extensive experiments on four multi-modality FL benchmarks with
different types of data heterogeneity, where FedDAT substantially outperforms
the existing centralized PEFT methods adapted for FL.