Due to their complex nature, it is hard to characterize the ways in which machine learning models can misbehave or be exploited when deployed. Recent work on adversarial examples, i.e. inputs with minor perturbations that result in substantially different model predictions, is helpful in evaluating the robustness of these models by exposing the adversarial scenarios where they fail. However, these malicious perturbations are often unnatural, not semantically meaningful, and not applicable to complicated domains such as language. In this paper, we propose a framework to generate natural and legible adversarial examples that lie on the data manifold, by searching in semantic space of dense and continuous data representation, utilizing the recent advances in generative adversarial networks. We present generated adversaries to demonstrate the potential of the proposed approach for black-box classifiers for a wide range of applications such as image classification, textual entailment, and machine translation. We include experiments to show that the generated adversaries are natural, legible to humans, and useful in evaluating and analyzing black-box classifiers. With the impressive success and extensive use of machine learning models in various securitysensitive applications, it has become crucial to study vulnerabilities in these systems. BID7 show that adversarial manipulations of input data often result in incorrect predictions from classifiers. This raises serious concerns regarding the security and integrity of existing machine learning algorithms, especially when even state-of-the-art models including deep neural networks have been shown to be highly vulnerable to adversarial attacks with intentionally worst-case perturbations to the input BID28 BID10 BID16 BID22 BID17 . These adversaries are generated effectively with access to the gradients of target models, resulting in much higher successful attack rates than data perturbed by random noise of even larger magnitude. Further, training models by including such adversaries can provide machine learning models with additional regularization benefits BID10 .Although these adversarial examples expose "blind spots" in machine learning models, they are unnatural, i.e. these worst-case perturbed instances are not ones the classifier is likely to face when deployed. Due to this , it is difficult to gain helpful insights into the fundamental decision behavior inside the black-box classifier: why is the decision different for the adversary, what can we change in order to prevent this behavior, and is the classifier robust to natural variations in the data when not in an adversarial scenario? Moreover, there is often a mismatch between the input space and the semantic space that we can understand. Changes to the input we may not think meaningful, like slight rotation or translation in images, often lead to substantial differences in the input instance. For example , BID24 show that minimal changes in the lighting conditions can fool automated-driving systems, a behavior adversarial examples are unable to discover. Due to the unnatural perturbations, these approaches cannot be applied to complex domains such as language, in which enforcing grammar and semantic similarity is difficult when perturbing instances. Therefore, existing approaches that find adversarial examples for text often result in ungrammatical sentences, as in the examples generated by BID19 , or require manual intervention, as in BID14 .In this paper , we introduce a framework to generate natural adversarial examples, i.e. instances that are meaningfully similar, valid/legible, and helpful for interpretation. The primary intuition Given an instance (a), existing FGSM approach BID10 adds small perturbations in (b) , that change the prediction of the model (to be "2", in this case).Instead of such random-looking noise, our framework generates natural adversarial examples, such as in (e), where the differences, shown in (d) (with blue/+, red/-), are meaningful changes to the strokes.behind our proposed approach is to perform the search for adversaries in a dense and continuous representation of the data instead of searching in the input data space directly. We use generative adversarial networks (GANs) to learn a projection to map normally distributed fixed-length vectors to data instances. Given an input instance, we search for adversaries in the neighborhood of its corresponding representation in latent space by sampling within a range that is recursively tightened. FIG0 provides an example of adversaries for digit recognition. Given a multi-layer perceptron (MLP) for MNIST and an image from test data FIG0 ), our approach generates a natural adversarial example ( FIG0 ) which is classified incorrectly as "2" by the classifier. Compared to the adversary generated by the existing Fast Gradient Sign Method (FGSM) BID10 that adds gradient-based noise FIG0 , our adversary ( FIG0 ) looks like a hand-written digit similar to the original input. Further, the difference ( FIG0 ) provides some insight into the classifier's behavior, such as the fact that slightly thickening (blue) the bottom stroke and thinning (red) the one above it, fools the classifier.We apply our approach to both image and text domains, and generate adversaries that are more natural and grammatical, semantically close to the input, and helpful to interpret the local behavior of black-box models. We present examples of natural adversaries for image classification, textual entailment, and machine translation. Experiments and human evaluation also demonstrate that our approach can help evaluate the robustness of black-box classifiers, even without labeled training data. Our framework builds upon GANs as the generative models, and thus the capabilities of GANs directly effects the quality of generated examples. In visual domains, although there have been lots of appealing results produced by GANs, the training is well known to be brittle. Many recent approaches address how to improve the training stability and the objective function of GANs BID27 . BID11 further improve the training of WGAN with regularization of gradient penalty instead of weight clipping. In our practice, we observe that we need to carefully balance the capacities of the generator, the critic, and the inverter that we introduced, to avoid situations such as model collapse. For natural languages, because of the discrete nature and non-differentiability, applications related to text generation have been relatively less studied. BID31 propose to incorporate a discrete structure autoencoder with continuous code space regularized by WGAN for text generation. Given that there are some concerns about whether GANs actually learn the distribution BID3 , it is worth noting that we can also incorporate other generative models such as Variational Auto-Encoders (VAEs) (Kingma & Welling, 2014) into our framework, as used in BID13 to generate text with controllable attributes, which we will explore in the future. We focus on GANs because adversarial training often results in higher quality images, while VAEs tend to produce blurrier ones BID8 . We also plan to apply the fusion and variant of VAEs and GANs such as α-GAN in BID26 and Wasserstein Auto-Encoders in BID29 . Note that as more advanced GANs are introduced to address these issues, they can be directly incorporated into our framework.Our iterative stochastic search algorithm for identifying adversaries is computationally expensive since it is based on naive sampling and local-search. Search based on gradients such as FGSM are not applicable to our setup because of black-box classifiers and discrete domain applications. We improve the efficiency with hybrid shrinking search by using a coarse-to-fine strategy that finds the upper-bounds by using fewer samples, and then performs finer search in the restricted range. We observe around 4× speedup with this search while achieving similar results as the iterative search. The accuracy of our inverter mapping the input to its corresponding dense vector in latent space is also important for searching adversaries in the right neighborhood. In our experiments, we find that fine-tuning the latent vector produced by the inverter with a fixed GAN can further refine the generated adversarial examples, and we will investigate other such extensions of the search in future.There is an implicit assumption in this work that the generated samples are within the same class if the added perturbations are small enough, and the generated samples look as if they belong to different classes when the perturbations are large. However, note that it is also the case for FGSM and other such approaches: when their is small, the noise is imperceptible; but with a large , one often finds noisy instances that might be in a different class (see TAB0 , digit 8 for an example). While we do observe this behavior in some cases, the corresponding classifiers require much more substantial changes to the input, which is why we can utilize our approach to evaluate black-box classifiers. In this paper, we propose a framework for generating natural adversaries against black-box classifiers, and apply the same approach to both visual and textual domains. We obtain adversaries that are legible, grammatical, and meaningfully similar to the input. We show that these natural adversaries can help in interpreting the decision behavior and evaluating the accuracy of black-box classifiers even in absence of labeled training data. We use our approach, built upon recent work in GANs, to generate adversaries for a wide range of applications including image classification, textual entailment, and machine translation (via the Google Translate API). Code used to generate such natural adversaries is available at https://github.com/zhengliz/natural-adversary.
We propose a framework to generate “natural” adversaries against black-box classifiers for both visual and textual domains, by doing the search for adversaries in the latent semantic space. Suggests a method for creation of semantical adversary examples. Proposes a framework to generate natural adversarial examples by searching adversaries in a latent space of dense and continuous data representation 