Corrupting Neuron Explanations of Deep Visual Features
Divyansh Srivastava Tuomas Oikarinen Tsui-Wei Weng
The inability of DNNs to explain their black-box behavior has led to a recent
surge of explainability methods. However, there are growing concerns that these
explainability methods are not robust and trustworthy. In this work, we perform
the first robustness analysis of Neuron Explanation Methods under a unified
pipeline and show that these explanations can be significantly corrupted by
random noises and well-designed perturbations added to their probing data. We
find that even adding small random noise with a standard deviation of 0.02 can
already change the assigned concepts of up to 28% neurons in the deeper layers.
Furthermore, we devise a novel corruption algorithm and show that our algorithm
can manipulate the explanation of more than 80% neurons by poisoning less than
10% of probing data. This raises the concern of trusting Neuron Explanation
Methods in real-life safety and fairness critical applications.