Counterfactual Image Generation for adversarially robust and
  interpretable Classifiers
Rafael Bischof Florian Scheidegger Michael A. Kraus A. Cristiano I. Malossi
Neural Image Classifiers are effective but inherently hard to interpret and
susceptible to adversarial attacks. Solutions to both problems exist, among
others, in the form of counterfactual examples generation to enhance
explainability or adversarially augment training datasets for improved
robustness. However, existing methods exclusively address only one of the
issues. We propose a unified framework leveraging image-to-image translation
Generative Adversarial Networks (GANs) to produce counterfactual samples that
highlight salient regions for interpretability and act as adversarial samples
to augment the dataset for more robustness. This is achieved by combining the
classifier and discriminator into a single model that attributes real images to
their respective classes and flags generated images as "fake". We assess the
method's effectiveness by evaluating (i) the produced explainability masks on a
semantic segmentation task for concrete cracks and (ii) the model's resilience
against the Projected Gradient Descent (PGD) attack on a fruit defects
detection problem. Our produced saliency maps are highly descriptive, achieving
competitive IoU values compared to classical segmentation models despite being
trained exclusively on classification labels. Furthermore, the model exhibits
improved robustness to adversarial attacks, and we show how the discriminator's
"fakeness" value serves as an uncertainty measure of the predictions.