Diffusion Theory as a Scalpel: Detecting and Purifying Poisonous
  Dimensions in Pre-trained Language Models Caused by Backdoor or Bias
Zhiyuan Zhang Deli Chen Hao Zhou Fandong Meng Jie Zhou Xu Sun
Pre-trained Language Models (PLMs) may be poisonous with backdoors or bias
injected by the suspicious attacker during the fine-tuning process. A core
challenge of purifying potentially poisonous PLMs is precisely finding
poisonous dimensions. To settle this issue, we propose the Fine-purifying
approach, which utilizes the diffusion theory to study the dynamic process of
fine-tuning for finding potentially poisonous dimensions. According to the
relationship between parameter drifts and Hessians of different dimensions, we
can detect poisonous dimensions with abnormal dynamics, purify them by
resetting them to clean pre-trained weights, and then fine-tune the purified
weights on a small clean dataset. To the best of our knowledge, we are the
first to study the dynamics guided by the diffusion theory for safety or
defense purposes. Experimental results validate the effectiveness of
Fine-purifying even with a small clean dataset.