Rethinking Adversarial Training with Neural Tangent Kernel
Guanlin Li Han Qiu Shangwei Guo Jiwei Li Tianwei Zhang
Adversarial training (AT) is an important and attractive topic in deep
learning security, exhibiting mysteries and odd properties. Recent studies of
neural network training dynamics based on Neural Tangent Kernel (NTK) make it
possible to reacquaint AT and deeply analyze its properties. In this paper, we
perform an in-depth investigation of AT process and properties with NTK, such
as NTK evolution. We uncover three new findings that are missed in previous
works. First, we disclose the impact of data normalization on AT and the
importance of unbiased estimators in batch normalization layers. Second, we
experimentally explore the kernel dynamics and propose more time-saving AT
methods. Third, we study the spectrum feature inside the kernel to address the
catastrophic overfitting problem. To the best of our knowledge, it is the first
work leveraging the observations of kernel dynamics to improve existing AT
methods.