FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language
  Models
Jingwei Sun Ziyue Xu Hongxu Yin Dong Yang Daguang Xu Yiran Chen Holger R. Roth
Pre-trained language models (PLM) have revolutionized the NLP landscape,
achieving stellar performances across diverse tasks. These models, while
benefiting from vast training data, often require fine-tuning on specific data
to cater to distinct downstream tasks. However, this data adaptation process
has inherent security and privacy concerns, primarily when leveraging
user-generated, device-residing data. Federated learning (FL) provides a
solution, allowing collaborative model fine-tuning without centralized data
collection. However, applying FL to finetune PLMs is hampered by challenges,
including restricted model parameter access, high computational requirements,
and communication overheads. This paper introduces Federated Black-box Prompt
Tuning (FedBPT), a framework designed to address these challenges. FedBPT does
not require the clients to access the model parameters. By focusing on training
optimal prompts and utilizing gradient-free optimization methods, FedBPT
reduces the number of exchanged variables, boosts communication efficiency, and
minimizes computational and storage costs. Experiments highlight the
framework's ability to drastically cut communication and memory costs while
maintaining competitive performance. Ultimately, FedBPT presents a promising
solution for efficient, privacy-preserving fine-tuning of PLM in the age of
large language models.