Hijacking Large Language Models via Adversarial In-Context Learning
Yao Qiang Xiangyu Zhou Dongxiao Zhu
In-context learning (ICL) has emerged as a powerful paradigm leveraging LLMs
for specific tasks by utilizing labeled examples as demonstrations in the
precondition prompts. Despite its promising performance, ICL suffers from
instability with the choice and arrangement of examples. Additionally, crafted
adversarial attacks pose a notable threat to the robustness of ICL. However,
existing attacks are either easy to detect, rely on external models, or lack
specificity towards ICL. To address these issues, this work introduces a novel
transferable attack for ICL, aiming to hijack LLMs to generate the targeted
response. The proposed LLM hijacking attack leverages a gradient-based prompt
search method to learn and append imperceptible adversarial suffixes to the
in-context demonstrations. Extensive experimental results on various tasks and
datasets demonstrate the effectiveness of our LLM hijacking attack, resulting
in a distracted attention towards adversarial tokens, consequently leading to
the targeted unwanted outputs.