In the adversarial-perturbation problem of neural networks, an adversary starts with a neural network model $F$ and a point $\bfx$ that $F$ classifies correctly, and applies a \emph{small perturbation} to  $\bfx$ to produce another point $\bfx'$ that $F$ classifies \emph{incorrectly}.  In this paper, we propose taking into account \emph{the inherent confidence information} produced by models when studying adversarial perturbations, where a natural measure of ``confidence'' is \|F(\bfx)\|_\infty$ (i.e. how confident $F$ is about its prediction?) . Motivated by a thought experiment based on the manifold assumption, we propose a ``goodness property'' of models which states that \emph{confident regions of a good model should be well separated}. We give formalizations of this property and examine existing robust training objectives in view of them. Interestingly, we find that a recent objective by Madry et al. encourages training a model that satisfies well our formal version of the goodness property, but has a weak control of points that are wrong but with low confidence. However, if Madry et al.'s model is indeed a good solution to their objective, then good and bad points are now distinguishable and we can try to embed uncertain points back to the closest confident region to get (hopefully) correct predictions. We thus propose embedding objectives and algorithms, and perform an empirical study using this method. Our experimental results are encouraging: Madry et al.'s model wrapped with our embedding procedure achieves almost perfect success rate in defending against attacks that the base model fails on, while retaining good generalization behavior.
 In the adversarial perturbation problem of neural networks, an adversary starts with a neural network model F and a point x that F classifies correctly (we assume that F ends with a softmax layer, which is common in the literature), and applies a small perturbation to x to produce another point x that F classifies incorrectly. BID17 first noticed the vulnerability of existing (deep) neural networks to adversarial perturbations, which is a somewhat surprising phenomenon given the great generalization capability of these networks. Since then, a line of research (see, for example, BID5 ; BID14 ; BID9 ; BID8 ) has been devoted to harden neural networks against adversarial perturbation. However, while modest progress has been made, until now there is still a large gap in successfully defending against more advanced attacks, such as attack by BID4 .In this paper, we propose taking into account the inherent confidence information produced by models when studying adversarial perturbations. To this end, a natural measure of confidence is F (x) ∞ (i.e., how confident F is about its prediction?) We motivate this consideration with a thought experiment based on the manifold assumption BID18 ; man) that is commonly made in unsupervised and semi-supervised learning, which states that natural data points lie on (or near to) separate low dimensional manifolds for different classes. Essentially , if one believes that deep neural networks learn to approximate well these low dimensional manifolds, then an ideal model should have the property that it can confidently distinguish points from natural manifolds as they are well separated due to the assumption. Moreover, since the learner never sees points that are far away from the natural manifolds, an ideal model should not claim confidence there.Taking this perspective, we propose a goodness property which states that confident regions of a good model should be well separated. We give formalizations of this property and examine existing robust training objectives in view of them. Interestingly, we find that a recent objective function by BID8 encourages training a model that satisfies well our formal version of the goodness property, in the sense that high-confidence predictions of different classes are well separated. On the other hand, our analysis also indicates that there could be many points with wrong but low-confidence predictions. Therefore, if Madry et al.' s model is indeed a good solution to their objective, then we can distinguish between good and bad points with confidence, and try to embed a low-confidence point back to confident regions to get (hopefully) the correct prediction.We propose two embedding objectives: (1) δ-Most Confident Neighbor (MCN δ ), where MCN δ (F, x) = arg max z∈N (x,δ) F (z) ∞ for a point x, a radius parameter δ > 0, and N (x, δ) the δ-neighborhood around x. (2) p-Nearest Confident Neighbor (NCN p ), where NCN p (F, x) = arg min z z − x subject to F (z) ∞ ≥ p, for a point x and a confidence parameter p ∈ (0, 1). With these, the end to end predictions become F (MCN δ (F, x)) and F (NCN p (F, x)). We note that these objectives are semantic: They fail only when the model has a confident but wrong prediction in the neighborhood.We perform an empirical study over CIFAR10. We first empirically validate that Madry et al.' s model is better, in view of our goodness property, than models trained without a robustness objective. We then give end-to-end defense results based on our method. Specifically, we propose using gradient based optimization, such as Carlini-Wagner attacks BID4 (which, however, are now used for defense) to solve MCN δ or NCN p . Our empirical results are encouraging : (1) It achieves almost perfect success rate in defending against attacks that the base model fails on, and (2) It also retains the good generalization behavior of the base model.The rest of the paper is organized as follows: We first discuss important prior work in Section 2 and some preliminaries in Section 3. Then Section 4 proposes the goodness property, and examines Madry et al.'s robust training objective function in view of the property. We then present embedding objectives and algorithms for handling low-confidence points in Section 5. Section 6 performs an empirical study where we validate that Madry et al.'s robust model satisfies well our goodness property, and then give defense results for our technique. Section 7 concludes with discussions on implications of our method. BID17 first observed the susceptibility of deep neural networks to adversarial perturbations. Since then, a large body of work have been devoted to studying hardening neural networks for this problem (a subset of work in this direction is BID5 BID14 ; BID9 ). Simultaneously, another line of work have been devoted to devise more effective or efficient attacks (a small set of work in this direction is BID10 BID13 BID4 ). Unfortunately, there still seems to be a large gap for the defense methods to defend against more sophisticated attacks, such as CarliniWagner attacks BID4 . For example, while the recent robust residual network constructed by BID8 achieves encouraging robustness results on MNIST, on CIFAR10 the accuracy against a strong adversary, such as attacks by BID4 , can be as low as 45.8%. We now discuss our method and results. (a) Base model on x, unconfident prediction "automobile" DISPLAYFORM0 (e) A valid "ship" that resembles the "automobile"(f ) Base model on x, unconfident prediction "airplane" ( g) CarliniWagnerShell on x, unconfident prediction "ship" (h) Base model on x , unconfident prediction "ship" (i) CarliniWagnerShell on x , unconfident prediction "airplane" (j) A valid "ship" image that resembles the "airplane" Figure 2 : The images (a) and (e) are the points where our CarliniWagnerShell makes a wrong prediction. We show the original image on the first column, CarliniWagnerShell perturbations of the original images on the second column, adversarial perturbed images on the third column, and CarliniWagnerShell perturbations of the adversarial examples on the fourth column. Example ship images are presented on the last row for comparison.Our technical contributions. Our first technical contribution is the proposal to take into account the inherent confidence information produced by models when studying adversarial perturbations. Our second technical contribution is a formulation of a goodness property of models, and an analysis of existing models in view of the property. Interestingly, and somewhat surprisingly, we find that a recent robust training objective function by BID8 encourages good separation of highconfidence points, but has essentially no control of low-confidence points. Our third contribution is the proposal of embedding to handle low-confidence points. Our final contribution is a first empirical study that validates Madry et al.'s model in terms of the goodness property, and further demonstrates that a good model, when wrapped with embedding, simultaneously achieves good generalization and almost perfect robustness.Interpretations of our results. One interpretation of our results is that adversarial perturbations can naturally coexist with good generalization. While this is manifested in our analysis of Madry et al.'s objective function, we think that this phenomenon naturally and generally exists if one takes again a manifold point of view: Since natural manifolds reside in low dimensions, it seems much more challenging to control the confidence boundary, where "adversarial perturbations" exist in abundance, than controlling separation of the learned structures (i.e. confident regions that approximate the underlying manifolds).On the other hand, if all one cares about is robustness (decision does not change in a small neighborhood), then adversarial perturbation problem can be resolved by combining goodness property with embedding. For example, consider the following "good model:" If a data point is from the training set, then it outputs the correct label with confidence 1, otherwise it outputs a uniform distribution over labels. In other words, this model learns nothing but fitting the training set. We note that in this case, the adversarial perturbation problem is only well defined around the training points. Moreover , embedding now becomes 1-nearest neighbor search among the training points. As a result , the model is still perfectly robust with our method if training points are well separated.Highly confident predictions on random noises. We note that several work shows that neural networks can have highly confident predictions on random noises (e.g., BID11 ). In view of our work it is somewhat not surprising that neural networks can have such behaviors: These points are essentially ones that are far away from the "universe" the learner is asked to learn, and so if we do not control the training of neural networks to not claim confidence over the structure it has never seen, then it is valid to fit a model that has good behaviors on natural manifolds but also divergent behaviors outside. After all, why is a network supposed to work on points that are far away from the underlying natural manifolds, which is essentially the data generating distribution? Finally, we note that the adversarial perturbation problem is not well defined even near those points.
Defending against adversarial perturbations of neural networks from manifold assumption  The manuscript proposes two objective functions based on the manifold assumption as defense mechanisms against adversarial examples. Defending against adversarial attacks based on the manifold assumption of natural data