The race to robustness: exploiting fragile models for urban camouflage
  and the imperative for machine learning security
Harriet Farlow Matthew Garratt Gavin Mount Tim Lynar
Adversarial Machine Learning (AML) represents the ability to disrupt Machine
Learning (ML) algorithms through a range of methods that broadly exploit the
architecture of deep learning optimisation. This paper presents Distributed
Adversarial Regions (DAR), a novel method that implements distributed
instantiations of computer vision-based AML attack methods that may be used to
disguise objects from image recognition in both white and black box settings.
We consider the context of object detection models used in urban environments,
and benchmark the MobileNetV2, NasNetMobile and DenseNet169 models against a
subset of relevant images from the ImageNet dataset. We evaluate optimal
parameters (size, number and perturbation method), and compare to
state-of-the-art AML techniques that perturb the entire image. We find that
DARs can cause a reduction in confidence of 40.4% on average, but with the
benefit of not requiring the entire image, or the focal object, to be
perturbed. The DAR method is a deliberately simple approach where the intention
is to highlight how an adversary with very little skill could attack models
that may already be productionised, and to emphasise the fragility of
foundational object detection models. We present this as a contribution to the
field of ML security as well as AML. This paper contributes a novel adversarial
method, an original comparison between DARs and other AML methods, and frames
it in a new context - that of urban camouflage and the necessity for ML
security and model robustness.