A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO
  and Toxicity
Andrew Lee Xiaoyan Bai Itamar Pres Martin Wattenberg Jonathan K. Kummerfeld Rada Mihalcea
While alignment algorithms are now commonly used to tune pre-trained language
models towards a user's preferences, we lack explanations for the underlying
mechanisms in which models become ``aligned'', thus making it difficult to
explain phenomena like jailbreaks. In this work we study a popular algorithm,
direct preference optimization (DPO), and the mechanisms by which it reduces
toxicity. Namely, we first study how toxicity is represented and elicited in a
pre-trained language model, GPT2-medium. We then apply DPO with a carefully
crafted pairwise dataset to reduce toxicity. We examine how the resulting model
averts toxic outputs, and find that capabilities learned from pre-training are
not removed, but rather bypassed. We use this insight to demonstrate a simple
method to un-align the model, reverting it back to its toxic behavior.