We augment adversarial training (AT) with worst case adversarial training
 (WCAT) which improves adversarial robustness by 11% over the current state-
 of-the-art result in the `2-norm on CIFAR-10. We interpret adversarial training as
 Total Variation Regularization, which is a fundamental tool in mathematical im-
 age processing, and WCAT as Lipschitz regularization, which appears in Image
 Inpainting. We obtain verifiable worst and average case robustness guarantees,
 based on the expected and maximum values of the norm of the gradient of the
 loss. We augment adversarial training (AT) with worst case adversarial training (WCAT) which improves adversarial robustness by 11% over the current state-of-the-art result BID27 in the 2 norm. The method also achieves results comparable to the state-of-the-art results of BID22 in the ∞ norm. Moreover, our adversarial training step uses only one gradient evaluation compared to seven steps in the BID22 work. The worst case adversarial training method is described as follows. During adversarial training, the gradient of the loss is computed for each perturbed image. WCAT records the largest of the gradients norms, and adds a penalty to the loss proportional to this term. In many cases we observe that models trained with AT and WCAT have improved test/validation error over the unregularized model.In §2 we show that the norm of the gradient of the loss of the model is a measure of the robustness of a model to adversarial examples. We obtain verifiable worst and average case robustness guarantees, based on the expected and maximum values of the norm of the gradient of the loss. We then compute these quantities empirically on trained models, and demonstrate that improving these quantities leads to proportional improvements in adversarial robustness.In §3 we interpret adversarial training as Total Variation (TV) Regularization, which is a fundamental tool in mathematical image processing. TV regularization was introduced for image denoising BID29 . It is a measure of the variation of a function, allowing for discontinuities. We also show that WCAT corresponds to Lipschitz regularization, which appears in Image Inpainting BID2 and function approximation BID7 BID25 . Lipschitz regularization was used in a recent proof of generalization of deep neural networks BID26 . Write (x) = • f (x) for the loss of the model. We show that training with AT and WCAT is equivalent to minimizing DISPLAYFORM0 where is the size of the adversarial training perturbation, and λ is the WCAT multiplier. The dual norm · * corresponds to · 1 for attacks measured in ∞ and to · 2 for attacks measured in 2 , 1 see §3.1.
Improvements to adversarial robustness, as well as provable robustness guarantees, are obtained by augmenting adversarial training with a tractable Lipschitz regularization Explores augmenting the training loss with an additional gradient regularization term to improve robustness of models against adversarial examples Uses a trick to simplify the adversarial loss by one in which the adversarial perturbation appears in closed form.