Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs
  Without Fine-Tuning
Adib Hasan Ileana Rugina Alex Wang
Large Language Models (LLMs) are vulnerable to `Jailbreaking' prompts, a type
of attack that can coax these models into generating harmful and illegal
content. In this paper, we show that pruning up to 20% of LLM parameters
markedly increases their resistance to such attacks without additional training
and without sacrificing their performance in standard benchmarks. Intriguingly,
we discovered that the enhanced safety observed post-pruning correlates to the
initial safety training level of the model, hinting that the effect of pruning
could be more general and may hold for other LLM behaviors beyond safety.
Additionally, we introduce a curated dataset of 225 harmful tasks across five
categories, inserted into ten different Jailbreaking prompts, showing that
pruning aids LLMs in concentrating attention on task-relevant tokens in
jailbreaking prompts. Lastly, our experiments reveal that the prominent chat
models, such as LLaMA-2 Chat, Vicuna, and Mistral Instruct exhibit high
susceptibility to jailbreaking attacks, with some categories achieving nearly
70-100% success rate. These insights underline the potential of pruning as a
generalizable approach for improving LLM safety, reliability, and potentially
other desired behaviors.