Test-time Backdoor Mitigation for Black-Box Large Language Models with
  Defensive Demonstrations
Wenjie Mo Jiashu Xu Qin Liu Jiongxiao Wang Jun Yan Chaowei Xiao Muhao Chen
Existing studies in backdoor defense have predominantly focused on the
training phase, overlooking the critical aspect of testing time defense. This
gap becomes particularly pronounced in the context of Large Language Models
(LLMs) deployed as Web Services, which typically offer only black-box access,
rendering training-time defenses impractical. To bridge this gap, our work
introduces defensive demonstrations, an innovative backdoor defense strategy
for blackbox large language models. Our method involves identifying the task
and retrieving task-relevant demonstrations from an uncontaminated pool. These
demonstrations are then combined with user queries and presented to the model
during testing, without requiring any modifications/tuning to the black-box
model or insights into its internal mechanisms. Defensive demonstrations are
designed to counteract the adverse effects of triggers, aiming to recalibrate
and correct the behavior of poisoned models during test-time evaluations.
Extensive experiments show that defensive demonstrations are effective in
defending both instance-level and instruction-level backdoor attacks, not only
rectifying the behavior of poisoned models but also surpassing existing
baselines in most scenarios.