Goodhart's Law Applies to NLP's Explanation Benchmarks
Jennifer Hsia Danish Pruthi Aarti Singh Zachary C. Lipton
Despite the rising popularity of saliency-based explanations, the research
community remains at an impasse, facing doubts concerning their purpose,
efficacy, and tendency to contradict each other. Seeking to unite the
community's efforts around common goals, several recent works have proposed
evaluation metrics. In this paper, we critically examine two sets of metrics:
the ERASER metrics (comprehensiveness and sufficiency) and the EVAL-X metrics,
focusing our inquiry on natural language processing. First, we show that we can
inflate a model's comprehensiveness and sufficiency scores dramatically without
altering its predictions or explanations on in-distribution test inputs. Our
strategy exploits the tendency for extracted explanations and their complements
to be "out-of-support" relative to each other and in-distribution inputs. Next,
we demonstrate that the EVAL-X metrics can be inflated arbitrarily by a simple
method that encodes the label, even though EVAL-X is precisely motivated to
address such exploits. Our results raise doubts about the ability of current
metrics to guide explainability research, underscoring the need for a broader
reassessment of what precisely these metrics are intended to capture.