Immunization against harmful fine-tuning attacks
Domenic Rosati Jan Wehner Kai Williams Łukasz Bartoszcze Jan Batzner Hassan Sajjad Frank Rudzicz
Approaches to aligning large language models (LLMs) with human values has
focused on correcting misalignment that emerges from pretraining. However, this
focus overlooks another source of misalignment: bad actors might purposely
fine-tune LLMs to achieve harmful goals. In this paper, we present an emerging
threat model that has arisen from alignment circumvention and fine-tuning
attacks. However, lacking in previous works is a clear presentation of the
conditions for effective defence. We propose a set of conditions for effective
defence against harmful fine-tuning in LLMs called "Immunization conditions,"
which help us understand how we would construct and measure future defences.
Using this formal framework for defence, we offer a synthesis of different
research directions that might be persued to prevent harmful fine-tuning
attacks and provide a demonstration of how to use these conditions
experimentally showing early results of using an adversarial loss to immunize
LLama2-7b-chat.