Efficient Model Personalization in Federated Learning via
  Client-Specific Prompt Generation
Fu-En Yang Chien-Yi Wang Yu-Chiang Frank Wang
Federated learning (FL) emerges as a decentralized learning framework which
trains models from multiple distributed clients without sharing their data to
preserve privacy. Recently, large-scale pre-trained models (e.g., Vision
Transformer) have shown a strong capability of deriving robust representations.
However, the data heterogeneity among clients, the limited computation
resources, and the communication bandwidth restrict the deployment of
large-scale models in FL frameworks. To leverage robust representations from
large-scale models while enabling efficient model personalization for
heterogeneous clients, we propose a novel personalized FL framework of
client-specific Prompt Generation (pFedPG), which learns to deploy a
personalized prompt generator at the server for producing client-specific
visual prompts that efficiently adapts frozen backbones to local data
distributions. Our proposed framework jointly optimizes the stages of
personalized prompt adaptation locally and personalized prompt generation
globally. The former aims to train visual prompts that adapt foundation models
to each client, while the latter observes local optimization directions to
generate personalized prompts for all clients. Through extensive experiments on
benchmark datasets, we show that our pFedPG is favorable against
state-of-the-art personalized FL methods under various types of data
heterogeneity, allowing computation and communication efficient model
personalization.