Dual-Personalizing Adapter for Federated Foundation Models
Yiyuan Yang Guodong Long Tao Shen Jing Jiang Michael Blumenstein
Recently, foundation models, particularly large language models (LLMs), have
demonstrated an impressive ability to adapt to various tasks by fine-tuning
large amounts of instruction data. Notably, federated foundation models emerge
as a privacy preservation method to fine-tune models collaboratively under
federated learning (FL) settings by leveraging many distributed datasets with
non-IID data. To alleviate communication and computation overhead,
parameter-efficient methods are introduced for efficiency, and some research
adapted personalization methods to federated foundation models for better user
preferences alignment. However, a critical gap in existing research is the
neglect of test-time distribution shifts in real-world applications. Therefore,
to bridge this gap, we propose a new setting, termed test-time personalization,
which not only concentrates on the targeted local task but also extends to
other tasks that exhibit test-time distribution shifts. To address challenges
in this new setting, we explore a simple yet effective solution to learn a
comprehensive foundation model. Specifically, a dual-personalizing adapter
architecture (FedDPA) is proposed, comprising a global adapter and a local
adapter for addressing test-time distribution shifts and personalization,
respectively. Additionally, we introduce an instance-wise dynamic weighting
mechanism to optimize the balance between the global and local adapters,
enhancing overall performance. The effectiveness of the proposed method has
been evaluated on benchmark datasets across different NLP tasks.