Last One Standing: A Comparative Analysis of Security and Privacy of
  Soft Prompt Tuning, LoRA, and In-Context Learning
Rui Wen Tianhao Wang Michael Backes Yang Zhang Ahmed Salem
Large Language Models (LLMs) are powerful tools for natural language
processing, enabling novel applications and user experiences. However, to
achieve optimal performance, LLMs often require adaptation with private data,
which poses privacy and security challenges. Several techniques have been
proposed to adapt LLMs with private data, such as Low-Rank Adaptation (LoRA),
Soft Prompt Tuning (SPT), and In-Context Learning (ICL), but their comparative
privacy and security properties have not been systematically investigated. In
this work, we fill this gap by evaluating the robustness of LoRA, SPT, and ICL
against three types of well-established attacks: membership inference, which
exposes data leakage (privacy); backdoor, which injects malicious behavior
(security); and model stealing, which can violate intellectual property
(privacy and security). Our results show that there is no silver bullet for
privacy and security in LLM adaptation and each technique has different
strengths and weaknesses.