Assessing Visually-Continuous Corruption Robustness of Neural Networks
  Relative to Human Performance
Huakun Shen Boyue Caroline Hu Krzysztof Czarnecki Lina Marsso Marsha Chechik
While Neural Networks (NNs) have surpassed human accuracy in image
classification on ImageNet, they often lack robustness against image
corruption, i.e., corruption robustness. Yet such robustness is seemingly
effortless for human perception. In this paper, we propose visually-continuous
corruption robustness (VCR) -- an extension of corruption robustness to allow
assessing it over the wide and continuous range of changes that correspond to
the human perceptive quality (i.e., from the original image to the full
distortion of all perceived visual information), along with two novel
human-aware metrics for NN evaluation. To compare VCR of NNs with human
perception, we conducted extensive experiments on 14 commonly used image
corruptions with 7,718 human participants and state-of-the-art robust NN models
with different training objectives (e.g., standard, adversarial, corruption
robustness), different architectures (e.g., convolution NNs, vision
transformers), and different amounts of training data augmentation. Our study
showed that: 1) assessing robustness against continuous corruption can reveal
insufficient robustness undetected by existing benchmarks; as a result, 2) the
gap between NN and human robustness is larger than previously known; and
finally, 3) some image corruptions have a similar impact on human perception,
offering opportunities for more cost-effective robustness assessments. Our
validation set with 14 image corruptions, human robustness data, and the
evaluation code is provided as a toolbox and a benchmark.