Predictability and Comprehensibility in Post-Hoc XAI Methods: A
  User-Centered Analysis
Anahid Jalali Bernhard Haslhofer Simone Kriglstein Andreas Rauber
Post-hoc explainability methods aim to clarify predictions of black-box
machine learning models. However, it is still largely unclear how well users
comprehend the provided explanations and whether these increase the users
ability to predict the model behavior. We approach this question by conducting
a user study to evaluate comprehensibility and predictability in two widely
used tools: LIME and SHAP. Moreover, we investigate the effect of
counterfactual explanations and misclassifications on users ability to
understand and predict the model behavior. We find that the comprehensibility
of SHAP is significantly reduced when explanations are provided for samples
near a model's decision boundary. Furthermore, we find that counterfactual
explanations and misclassifications can significantly increase the users
understanding of how a machine learning model is making decisions. Based on our
findings, we also derive design recommendations for future post-hoc
explainability methods with increased comprehensibility and predictability.