Investigating Human-Identifiable Features Hidden in Adversarial
  Perturbations
Dennis Y. Menn Tzu-hsun Feng Sriram Vishwanath Hung-yi Lee
Neural networks perform exceedingly well across various machine learning
tasks but are not immune to adversarial perturbations. This vulnerability has
implications for real-world applications. While much research has been
conducted, the underlying reasons why neural networks fall prey to adversarial
attacks are not yet fully understood. Central to our study, which explores up
to five attack algorithms across three datasets, is the identification of
human-identifiable features in adversarial perturbations. Additionally, we
uncover two distinct effects manifesting within human-identifiable features.
Specifically, the masking effect is prominent in untargeted attacks, while the
generation effect is more common in targeted attacks. Using pixel-level
annotations, we extract such features and demonstrate their ability to
compromise target models. In addition, our findings indicate a notable extent
of similarity in perturbations across different attack algorithms when averaged
over multiple models. This work also provides insights into phenomena
associated with adversarial perturbations, such as transferability and model
interpretability. Our study contributes to a deeper understanding of the
underlying mechanisms behind adversarial attacks and offers insights for the
development of more resilient defense strategies for neural networks.