How adversarial attacks can disrupt seemingly stable accurate
  classifiers
Oliver J. Sutton Qinghua Zhou Ivan Y. Tyukin Alexander N. Gorban Alexander Bastounis Desmond J. Higham
Adversarial attacks dramatically change the output of an otherwise accurate
learning system using a seemingly inconsequential modification to a piece of
input data. Paradoxically, empirical evidence indicates that even systems which
are robust to large random perturbations of the input data remain susceptible
to small, easily constructed, adversarial perturbations of their inputs. Here,
we show that this may be seen as a fundamental feature of classifiers working
with high dimensional input data. We introduce a simple generic and
generalisable framework for which key behaviours observed in practical systems
arise with high probability -- notably the simultaneous susceptibility of the
(otherwise accurate) model to easily constructed adversarial attacks, and
robustness to random perturbations of the input data. We confirm that the same
phenomena are directly observed in practical neural networks trained on
standard image classification problems, where even large additive random noise
fails to trigger the adversarial instability of the network. A surprising
takeaway is that even small margins separating a classifier's decision surface
from training and testing data can hide adversarial susceptibility from being
detected using randomly sampled perturbations. Counterintuitively, using
additive noise during training or testing is therefore inefficient for
eradicating or detecting adversarial examples, and more demanding adversarial
training is required.