Sanity Checks for Explanation Uncertainty
Matias Valdenegro-Toro Mihir Mulye
Explanations for machine learning models can be hard to interpret or be
wrong. Combining an explanation method with an uncertainty estimation method
produces explanation uncertainty. Evaluating explanation uncertainty is
difficult. In this paper we propose sanity checks for uncertainty explanation
methods, where a weight and data randomization tests are defined for
explanations with uncertainty, allowing for quick tests to combinations of
uncertainty and explanation methods. We experimentally show the validity and
effectiveness of these tests on the CIFAR10 and California Housing datasets,
noting that Ensembles seem to consistently pass both tests with Guided
Backpropagation, Integrated Gradients, and LIME explanations.