Deep neural networks are vulnerable to adversarial examples, which can mislead classifiers by adding imperceptible perturbations. An intriguing property of adversarial examples is their good transferability, making black-box attacks feasible in real-world applications. Due to the threat of adversarial attacks, many methods have been proposed to improve the robustness, and several state-of-the-art defenses are shown to be robust against transferable adversarial examples. In this paper, we identify the attention shift phenomenon, which may hinder the transferability of adversarial examples to the defense models. It indicates that the defenses rely on different discriminative regions to make predictions compared with normally trained models. Therefore, we propose an attention-invariant attack method to generate more transferable adversarial examples. Extensive experiments on the ImageNet dataset validate the effectiveness of the proposed method. Our best attack fools eight state-of-the-art defenses at an 82% success rate on average based only on the transferability, demonstrating the insecurity of the defense techniques. Recent progress in machine learning and deep neural networks has led to substantial improvements in various pattern recognition tasks such as image understanding BID21 BID9 , speech recognition BID7 , and machine translation . However, deep neural networks are highly vulnerable to adversarial examples BID2 BID24 BID6 . They are maliciously generated by adding small perturbations to legitimate examples, but make deep neural networks produce unreasonable predictions. The existence of adversarial examples, even in the physical world BID11 BID5 BID1 , has raised concerns in security-sensitive applications, e.g., self-driving cars, healthcare and finance.Attacking deep neural networks has drawn an increasing attention since the generated adversarial examples can serve as a surrogate to evaluate the robustness of different models BID3 and help to improve the robustness BID6 BID16 . Several methods have been proposed to generate adversarial examples with the knowledge of the gradient information of a given model, such as fast gradient sign method BID6 , basic iterative method BID11 , and BID3 's method, which are known as white-box attacks. Moreover, it is shown that adversarial examples have cross-model transferability BID15 , i.e., the adversarial examples crafted for one model can fool a different model with a high probability. The transferability of adversarial examples enables practical black-box attacks to real-world applications and induces serious security issues.The threat of adversarial examples has motivated extensive research on building robust models or techniques to defend against adversarial attacks. These include training with adversarial examples BID6 BID12 BID27 BID16 , image denoising/transformation BID29 BID8 , leveraging generative models to move adversarial examples towards data manifold BID20 , and theoretically-certified defenses BID19 BID28 . Although the non-certified defenses have demonstrated robustness against common attacks, they do so by causing obfuscated gradients, which can be easily circumvented by new attacks BID0 . However, some of the defenses BID27 BID29 ; BID8 Figure 1: Demonstration of the attention shift phenomenon of the defense models compared with normally trained models. We adopt class activation mapping (Zhou et al., 2016) to visualize the attentive regions of three normally trained models-Inception v3 BID25 , Inception ResNet v2 BID26 , ResNet 152 BID9 and four defense models BID27 BID29 BID8 . These defense models focus their attention on slightly different regions compared with normally trained models, which may affect the transferability of adversarial examples. BID8 claim to be resistant to transferable adversarial examples, making black-box attacks difficult to evade these defenses.In this paper, we identify attention shift, that the defenses make predictions based on slightly different discriminative regions compared with normally trained models, as a phenomenon which may hinder the transferability of adversarial examples to the defense models. For example, we show the attention maps of several normally trained models and defense models in Fig. 1 , to represent the discriminative regions for their predictions. It is apparent that the normally trained models have similar attention maps while the defenses induce shifting attention maps. The attention shift of the defenses is caused by either training under different data distributions BID27 or transforming the inputs before classification BID29 BID8 . Therefore, the transferability of adversarial examples is largely reduced to the defenses since the structure information hidden in adversarial perturbations may be easily overlooked if a model focuses its attention on different regions.To mitigate the effect of attention shift and evade the defenses by transferable adversarial examples, we propose an attention-invariant attack method. In particular, we generate an adversarial example for an ensemble of examples composed of an legitimate one and its shifted versions. Therefore the resultant adversarial example is less sensitive to the attentive region of the white-box model being attacked and may have a bigger chance to fool another black-box model with a defense mechanism based on attention shift. We further show that this method can be simply implemented by convolving the gradient with a pre-defined kernel under a mild assumption. The proposed method can be integrated into any gradient-based attack methods such as fast gradient sign method and basic iterative method. Extensive experiments demonstrate that the proposed attention-invariant attack method helps to improve the success rates of black-box attacks against the defense models by a large margin. Our best attack reaches an average success rate of 82% to evade eight state-of-the-art defenses based only on the transferability, thus demonstrating the insecurity of the current defenses. In this paper, we propose an attention-invariant attack method to mitigate the attention shift phenomenon and generate more transferable adversarial examples against the defense models. Our method optimizes an adversarial image by using a set of shifted images. Based on an assumption, our method is simply implemented by convolving the gradient with a pre-defined kernel, and can be integrated into any gradient-based attack methods. We conduct experiments to validate the effectiveness of the proposed method. Our best attack A-DIM, the combination of the proposed attentioninvariant method and diverse input iterative method BID30 , can fool eight state-of-the-art defenses at an 82% success rate on average, where the adversarial examples are generated against four normally trained models. The results identify the vulnerability of the current defenses, which raises security issues for the development of more robust deep learning models.
We propose an attention-invariant attack method to generate more transferable adversarial examples for black-box attacks, which can fool state-of-the-art defenses with a high success rate. The paper proposes a new way of overcoming state of the art defences against adversarial attacks on CNN. This paper suggests that "attention shift" is a key property behind failure of adversarial attacks to transfer and propose an attention-invariant attack method