We introduce Explainable Adversarial Learning, ExL, an approach for training neural networks that are intrinsically robust to adversarial attacks. We find that the implicit generative modeling of random noise with the same loss function used during posterior maximization, improves a model's understanding of the data manifold furthering adversarial robustness. We prove our approach's efficacy and provide a simplistic visualization tool for understanding adversarial data, using Principal Component Analysis. Our analysis reveals that adversarial robustness, in general, manifests in models with higher variance along the high-ranked principal components. We show that models learnt with our approach perform remarkably well against a wide-range of attacks. Furthermore, combining ExL with state-of-the-art adversarial training extends the robustness of a model, even beyond what it is adversarially trained for, in both white-box and black-box attack scenarios. Despite surpassing human performance on several perception tasks, Machine Learning (ML) models remain vulnerable to adversarial examples: slightly perturbed inputs that are specifically designed to fool a model during test time BID2 BID29 BID6 BID23 . Recent works have demonstrated the security danger adversarial attacks pose across several platforms with ML backend such as computer vision BID29 BID6 BID22 BID15 BID19 , malware detectors BID16 BID33 BID7 BID10 and gaming environments BID11 BID1 . Even worse, adversarial inputs transfer across models: same inputs are misclassified by different models trained for the same task, thus enabling simple Black-Box (BB) 1 attacks against deployed ML systems .Several works BID14 BID24 BID3 demonstrating improved adversarial robustness have been shown to fail against stronger attacks BID0 . The state-of-the-art approach for BB defense is ensemble adversarial training that augments the training dataset of the target model with adversarial examples transferred from other pre-trained models BID30 . BID21 showed that models can even be made robust to White-Box (WB) 1 attacks by closely maximizing the model's loss with Projected Gradient Descent (PGD) based adversarial training. Despite this progress, errors still appear for perturbations beyond what the model is adversarially trained for BID27 .There have been several hypotheses explaining the susceptibility of ML models to such attacks. The most common one suggests that the overly linear behavior of deep neural models in a high dimensional input space causes adversarial examples BID6 BID20 . Another hypothesis suggests that adversarial examples are off the data manifold BID28 BID18 . Combining the two, we infer that excessive linearity causes models to extrapolate their behavior beyond the data manifold yielding pathological results for slightly perturbed inputs. A question worth asking here is: Can we improve the viability of the model to generalize better on such out-of-sample data?In this paper, we propose Explainable Adversarial Learning (ExL), wherein we introduce multiplicative noise into the training inputs and optimize it with Stochastic Gradient Descent (SGD) while minimizing the overall cost function over the training data. Essentially, the input noise (randomly initialized at the beginning) is gradually learnt during the training procedure. As a result, the noise approximately models the input distribution to effectively maximize the likelihood of the class labels given the inputs. FIG0 shows the input noise learnt during different stages of training by a simple convolutional network (ConvN et2 architecture discussed in Section 3 below), learning handwritten digits from MNIST dataset BID17 . We observe that the noise gradually transforms and finally assumes a shape that highlights the most dominant features in the MNIST training data. For instance, the MNIST images are centered digits on a black background. Noise, in fact, learnt this centered characteristic . This suggests that the model not only finds the right prediction but also the right explanation. Noise inculcates this explainable behavior by discovering some knowledge about the input/output distribution during training. FIG0 shows the noise learnt with ExL on colored CIFAR10 images BID13 ) (on ResNet18 architecture BID8 ), which reveals that noise template (also RGB) learns prominent color blobs on a greyish-black background, that de-emphasizes background pixels. A recent theory BID4 suggests that adversarial examples (off manifold misclassified points) occur in close proximity to randomly chosen inputs on the data manifold that are, in fact, correctly classified. With ExL, we hypothesize that the model learns to look in the vicinity of the onmanifold data points and thereby incorporate more out-of-sample data (without using any direct data augmentation) that, in turn, improves its generalization capability in the off-manifold input space. We empirically evaluate this hypothesis by visualizing and studying the relationship between the adversarial and the clean inputs using Principal Component Analysis (PCA). Examining the intermediate layer's output, we discover that models exhibiting adversarial robustness yield significantly lower distance between adversarial and clean inputs in the Principal Component (PC) subspace.We further harness this result to establish that ExL noise modeling, indeed, acquires an improved realization of the input/output distribution characteristics that enables it to generalize better. To further substantiate our hypothesis, we also show that ExL globally reduces the dimensionality of the space of adversarial examples BID31 . We evaluate our approach on classification tasks such as MNIST, CIFAR10 and CIFAR100 and show that models trained with ExL are extensively more adversarially robust. We also show that combining ExL with ensemble/PGD adversarial training significantly extends the robustness of a model, even beyond what it is adversarially trained for, in both BB/WB attack scenarios. We proposed Explainable Adversarial Learning, ExL, as a reliable method for improving adversarial robustness. Specifically, our key findings are: 1) We show that noise modeling at the input during discriminative training improves a model's ability to generalize better for out-of-sample adversarial data (without explicit data augmentation). 2) Our PCA variance and cosine distance analysis provides a significant perspective to visualize and quantify a model's response to adversarial/clean data.A crucial question one can ask is, How to break ExL defense? The recent work BID0 shows that many defense methods cause 'gradient masking' that eventually fail. We reiterate that, ExL alone does not give a strong BB/WB defense. However, the smoothening effect of noise modeling on the loss FIG3 suggests that noise modeling decreases the magnitude of the gradient masking effect. ExL does not change the classification model that makes it easy to be scaled to larger datasets while integrating with other adversarial defense techniques. Coupled with other defense, ExL performs remarkably (even for larger values). We combine ExL with EnsAdv & PGDAdv, which do not cause obfuscated gradients and hence can withstand strong attacks, however, upto a certain point. For WB perturbations much greater than the training value, ExL+PGDAdv also breaks. In fact, for adaptive BB adversaries BID30 or adversaries that query the model to yield full prediction confidence (not just the label), ExL+EnsAdv will be vulnerable. Note, advantage with ExL is, being independent of the attack/defense method, ExL can be potentially combined with stronger attacks developed in future, to create stronger defenses.While variance and principal subspace analysis help us understand a model's behavior, we cannot fully describe the structure of the manifold learnt by the linear subspace view. However, PCA does provide a basic intuition about the generalization capability of complex image models. In fact, our PC results establish the superiority of adversarial training methods (SGD ens ; SGD P GD : BID30 ; BID21 and can be used as a valid metric to gauge adversarial susceptibility in future proposals. Finally, as our likelihood theory (Eqn.1) indicates, better noise modeling techniques with improved gradient penalties can further improve robustness and requires further investigation. Also, performing noise modeling at intermediate layers to improve variance/explainability, and hence robustness, are other future work directions.A APPENDIX A: JUSTIFICATION OF X + N VS X × N AND USE OF ∇L N ≤ 0 FOR NOISE MODELING FIG0 : For MNIST dataset, we show the noise template learnt when we use multiplicative/additive noise (N ) for Explainable Learning. The final noise-integrated image (for a sample digit '9') that is fed to the network before and after training is also shown. Additive noise disrupts the image drastically. Multiplicative noise, on the other hand, enhances the relevant pixels while eliminating the background. Accuracy corrsponding to each scenario is also shown and compared against standard SGD training scenario (without any noise). Here, we train a simple convolutional architecture (ConvNet: 10C-M-20C-M-320FC) of 2 Convolutional (C) layers with 10, 20 filters, each followed by 2×2 Max-pooling (M) and a Fully-Connected (FC) layer of size 320. We use mini-batch SGD with momentum of 0.5, learning rate (η=0.1) decayed by 0.1 every 15 epochs and batchsize 64 to learn the network parameters. We trained 3 ConvNet models independently corresponding to each scenario for 30 epochs. For the ExL scenarios, we conduct noise modelling with only negative loss gradients (∇LN ≤ 0) with noise learning rate, ηnoise = 0.001, throughout the training process. Note, the noise image shown is the average across all 64 noise templates. Figure A2 : Here, we showcase the noise learnt by a simple convolutional network (ConvNet: 10C-M-20C-M-320FC), learning the CIFAR10 data with ExL (multiplicative noise) under different gradient update conditions. As with MNIST ( FIG0 , we observe that the noise learnt enhances the region of interest while deemphasizing the background pixels. Note, the noise in this case has RGB components as a result of which we see some prominent color blobs in the noise template after training. The performance table shows that using only negative gradients (i.e. ∇LN ≤ 0) during backpropagation for noise modelling yields minimal loss in accuracy as compared to a standard SGD trained model. We use mini-batch SGD with momentum of 0.9, weight decay 5e-4, learning rate (η=0.01) decayed by 0.2 every 10 epochs and batch-size 64 to learn the network parameters. We trained 4 ConvNet models independently corresponding to each scenario for 30 epochs. For the ExL scenarios, we conduct noise modelling by backpropagating the corresponding gradient with noise learning rate (ηnoise = 0.001) throughout the training process. Note, the noise image shown is the average across all 64 noise templates. We observe that ExL noise increases the explainability (or variance) along the high rank PCs. Also, as we go deeper into the network, the absolute difference of the variance values between SGD/ExL decreases. This is expected as the contribution of input noise on the overall representations decreases as we go deeper into the network. Moreover, there is a generic-to-specific transition in the hierarchy of learnt features of a deep neural network. Thus, the linear PC subspace analysis to quantify a model's knowledge of the data manifold is more applicable in the earlier layers, since they learn more general input-related characteristics. Nonetheless, we see that ExL model yields widened explainability than SGD for each intermediate layer except the final Block4 that feeds into the output layer. We use mini-batch SGD with momentum of 0.9, weight decay 5e-4, learning rate (η=0.1) decayed by 0.1 every 30 epochs and batch-size 64 to learn the network parameters. We trained 2 ResNet-18 models independently corresponding to each scenario for 60 epochs. For noise modelling, we use ηnoise = 0.001 decayed by 0.1 every 30 epochs. Note, we used a sample set of 700 test images to conduct the PCA. FIG2 : Here, we show the variance captured in the leading Principal Component (PC) dimensions for the Conv1 and Block1 learnt activations in response to both clean and adversarial inputs for ResNet-18 models correponding to the scenarios discussed in FIG1 . The model's variance for both clean and adversarial inputs are exactly same in case of ExL/SGD for Conv1 layers. For Block1, the adversarial input variance is slighlty lower in case of SGD than that of clean input. With ExL, the variance is still the same for Block1. This indicates that PC variance statistics cannot differentiate between a model's knowledge of on-/off-manifold data. It only tells us whether a model's underlying representation has acquired more knowledge about the data manifold. To analyze a model's understanding of adversarial data, we need to look into the relationship between the clean and adversarial projection onto the PC subspace and measure the cosine distance. Note, we used the Fast Gradient Sign Method (FGSM) method BID6 to create BB adversaries with a step size of 8/255, from another independently trained ResNet-18 model (source) with standard SGD. The source attack model has the same hyperparameters as the SGD model in FIG1 and is trained for 40 epochs.
Noise modeling at the input during discriminative training improves adversarial robustness. Propose PCA based evaluation metric for adversarial robustness This paper proposes, ExL, an adversarial training method using multiplicate noise that is shown to be helpful in defending against blackbox attacks on three datasets. This paper includes multiplicative noise N in training data to achieve adversarial robustness, when training on both model parameters theta and on the noise itself.