Revisiting and Exploring Efficient Fast Adversarial Training via LAW:
  Lipschitz Regularization and Auto Weight Averaging
Xiaojun Jia Yuefeng Chen Xiaofeng Mao Ranjie Duan Jindong Gu Rong Zhang Hui Xue Xiaochun Cao
Fast Adversarial Training (FAT) not only improves the model robustness but
also reduces the training cost of standard adversarial training. However, fast
adversarial training often suffers from Catastrophic Overfitting (CO), which
results in poor robustness performance. Catastrophic Overfitting describes the
phenomenon of a sudden and significant decrease in robust accuracy during the
training of fast adversarial training. Many effective techniques have been
developed to prevent Catastrophic Overfitting and improve the model robustness
from different perspectives. However, these techniques adopt inconsistent
training settings and require different training costs, i.e, training time and
memory costs, leading to unfair comparisons. In this paper, we conduct a
comprehensive study of over 10 fast adversarial training methods in terms of
adversarial robustness and training costs. We revisit the effectiveness and
efficiency of fast adversarial training techniques in preventing Catastrophic
Overfitting from the perspective of model local nonlinearity and propose an
effective Lipschitz regularization method for fast adversarial training.
Furthermore, we explore the effect of data augmentation and weight averaging in
fast adversarial training and propose a simple yet effective auto weight
averaging method to improve robustness further. By assembling these techniques,
we propose a FGSM-based fast adversarial training method equipped with
Lipschitz regularization and Auto Weight averaging, abbreviated as FGSM-LAW.
Experimental evaluations on four benchmark databases demonstrate the
superiority of the proposed method over state-of-the-art fast adversarial
training methods and the advanced standard adversarial training methods.