The Duet of Representations and How Explanations Exacerbate It
Charles Wan Rodrigo Belo Leid Zejnilović Susana Lavado
An algorithm effects a causal representation of relations between features
and labels in the human's perception. Such a representation might conflict with
the human's prior belief. Explanations can direct the human's attention to the
conflicting feature and away from other relevant features. This leads to causal
overattribution and may adversely affect the human's information processing. In
a field experiment we implemented an XGBoost-trained model as a decision-making
aid for counselors at a public employment service to predict candidates' risk
of long-term unemployment. The treatment group of counselors was also provided
with SHAP. The results show that the quality of the human's decision-making is
worse when a feature on which the human holds a conflicting prior belief is
displayed as part of the explanation.