While deep learning has led to remarkable results on a number of challenging problems, researchers have discovered a vulnerability of neural networks in adversarial settings, where small but carefully chosen perturbations to the input can make the models produce extremely inaccurate outputs. This makes these models particularly unsuitable for safety-critical application domains (e.g. self-driving cars) where robustness is extremely important. Recent work has shown that augmenting training with adversarially generated data provides some degree of robustness against test-time attacks. In this paper we investigate how this approach scales as we increase the computational budget given to the defender. We show that increasing the number of parameters in adversarially-trained models increases their robustness, and in particular that ensembling smaller models while adversarially training the entire ensemble as a single model is a more efficient way of spending said budget than simply using a larger single model. Crucially, we show that it is the adversarial training of the ensemble, rather than the ensembling of adversarially trained models, which provides robustness. Deep neural networks have demonstrated state-of-the-art performance in a wide range of application domains BID13 . However, researchers have discovered that deep networks are in some sense 'brittle', in that small changes to their inputs can result in wildly different outputs BID8 BID10 BID27 . For instance, practically imperceptible (to human) modifications to images can result in misclassification of the image with high confidence. Not only are networks susceptible to these 'attacks', but these attacks are also relatively easy to compute using standard optimization techniques BID5 BID6 . These changes are often referred to as adversarial perturbations, in the sense that an adversary could craft a very small change to the input in order to create an undesirable outcome. This phenomenon is not unique to image classification, nor to particular network architectures, nor to particular training algorithms BID23 .Adversarial attacks can be broken into different categories depending on how much knowledge of the underlying model the adversary has access to. In 'white-box ' attacks the adversary has full access to the model, and can perform both forward and backwards passes (though not change the weights or logic of the network) BID4 BID6 . In the 'black-box ' setting the adversary has no access to the model, but perhaps knows the dataset that the model was trained on BID23 . Despite several recent papers demonstrating new defences against adversarial attacks BID1 BID7 BID16 BID26 BID29 BID31 BID32 BID33 , recent papers have demonstrated that most of these new defences are still susceptible to attacks and largely just obfuscate the gradients that the attacker can follow, and that non-gradient based attacks are still effective BID30 ; BID2 . In this section we briefly discuss the possible reasons for the behaviours observed. As we saw, an ensemble of models trained adversarially outperforms the other setups at test time. We suspect, that this might be happening due to a mechanism described below.When the model is being trained, it is exposed to pairs of images, both "clean" and adversarially modified. The adversarial training exploits the fact that the original image is close to the decision boundary of the model. The model then, when provided with both clean and adversarial image would attempt to modify the decision boundary in order to engulf them both. It is relatively easy to imagine why SingleAdv would be weaker then the other models-it simply has less parameters than the competition. In order to accommodate the adversarial example it has to compromise the decision boundary somewhere else, pulling it close to other clean images, making it vulnerable to subsequent attack. This is illustrated in Figure 3a .The possible reason why Ensemble2Adv outperforms DoubleAdv is more elusive. Both models have the same number of parameters, so one could expect them to display a similar performance. As Ensemble2Adv is more robust to white box attack during test time we argue, that this might be due to the fact that in abundance of flexibility DoubleAdv tends often to spread out thin "tentacles" Figure 3: Different responses of various architectures to adversarial training. Solid lines represent decision boundary of the models that see only "clean" images. Dashed lines are the boundaries modified due to the presence of adversarial training. The black dot is a clean image, the red dot is its adversarial modification. In the presence of two models Model 1 is blue and Model 2 is green. In case it needs to be specified with respect to which model the adversarial example is constructed the red dot has a circle in an appropriate color around it.( Figure 3b ) which do not cover up too much of a space. On the other hand, given that Ensemble2Adv is comprised of two separate models they are both subject to lesser ability to overfit following from the smaller number of parameters available in them. Thus we argue, that in most cases the modification of the model with adversarial training covers the adversarial example by modifying one model more than the other. This way the decision boundary of the model modified to a lesser degree still "provides protection" for "clean" images, while at the same time the "tentacle" generated by the model modified more is thicker than the one DoubleAdv creates. We illustrate that with Figure 3c .Finally, it was shown that SeparateEnsemble2Adv is outperformed by a Ensemble2Adv trained "jointly". We think that this is due to the fact that the adversarial training has to weaken both of the submodels simultaneously. Figure 3d illustrates that.For a further illustration of the effects of adversarial training we plotted actual images of the decision boundaries for non-adversarially and adversarially trained Baseline and 2-Ensemble models (Figure 4) . DISPLAYFORM0 Figure 4: Decision boundaries for various architectures/training methods. Each column shows the decision regions of two models on the same 2-dimensional plane in the space of all images. On every picture the black dot corresponds to the datapoint-an unaltered (ship ) image from the test dataset. The light rectangle superimposed over the dot represents the bounds of the permitted attack region within the region. The two red arrows are the two vectors-attack directions on the base image, with respect to respectively the first and second tested model. The red dots are the images resulting from the attack. The plane presented is then the ( unique) 2-dimensional plane containing those 3 points. Dark grey is void (outside the slice boundaries), and all other pixels are generated by a forward pass of the model at those coordinates, with the colour used representing the majority class.Decision regions of the models are 3072-dimensional sets, so visualizing them itself poses a challenge. What we present are color-coded values of the models restricted to 2-dimensional planes in the space of all images, chosen so that the original image and the closest adversarial example (or attempt to find one) for both models in a pair being compared are co-planar. We observe, amongst other things, some support for the hypothesis put forward in Figure 3 : adversarial training adds "thickness" around the natural image points, pushing the boundary further away from them, and in doing so, making adversarial examples harder to find (even within the test set); ensembling makes some classes more "consistent" within the decision plane, but introduce small "pockets" or "tentacles" of other classes; and the combinator thereof removes said pockets to create large regions of the correct class around images. We believe that such an approach of choosing a good plane and plotting the values of models on it is a more informative way of visualizing phenomena taking place in the universe of robustness and adversarial examples than more traditional approaches like t-SNE plots BID18 . In this paper, we provide an empirical study of the effect of increasing the number of parameters in a model trained with adversarial training methods, with regard to its robustness to test-time adversarial attacks. We showed that while increasing parameters improves robustness, it is better to do so by ensembling smaller models than by producing one larger model. Through our experiments, we show that this result is not only due to ensembling alone, or to the implicit robustness of an ensemble of adversarially trained models, but specifically to due to the adversarial training of an ensemble as if it were a single model. We proposed a high level interpretation of why this phenomenon might occur. Further work should seek to determine whether scaling the number of models in the ensemble while controlling for number of parameters produces significant improvements over the minimal ensembles studied here in an attempt to draw conclusions about why such architectures are generally more robust than larger single models, even under adversarial training.
Adversarial training of ensembles provides robustness to adversarial examples beyond that observed in adversarially trained models and independently-trained ensembles thereof.  Proposes to train an ensemble of models jointly, where at each time step, a set of examples that are adversarial for the ensemble itself is incorporated in the learning.