A Split-and-Privatize Framework for Large Language Model Fine-Tuning
Xicong Shen Yang Liu Huiqi Liu Jue Hong Bing Duan Zirui Huang Yunlong Mao Ye Wu Di Wu
Fine-tuning is a prominent technique to adapt a pre-trained language model to
downstream scenarios. In parameter-efficient fine-tuning, only a small subset
of modules are trained over the downstream datasets, while leaving the rest of
the pre-trained model frozen to save computation resources. In recent years, a
popular productization form arises as Model-as-a-Service (MaaS), in which
vendors provide abundant pre-trained language models, server resources and core
functions, and customers can fine-tune, deploy and invoke their customized
model by accessing the one-stop MaaS with their own private dataset. In this
paper, we identify the model and data privacy leakage risks in MaaS
fine-tuning, and propose a Split-and-Privatize (SAP) framework, which manage to
mitigate the privacy issues by adapting the existing split learning
architecture. The proposed SAP framework is sufficiently investigated by
experiments, and the results indicate that it can enhance the empirical privacy
by 62% at the cost of 1% model performance degradation on the Stanford
Sentiment Treebank dataset.