Error Discovery by Clustering Influence Embeddings
Fulton Wang Julius Adebayo Sarah Tan Diego Garcia-Olano Narine Kokhlikyan
We present a method for identifying groups of test examples -- slices -- on
which a model under-performs, a task now known as slice discovery. We formalize
coherence -- a requirement that erroneous predictions, within a slice, should
be wrong for the same reason -- as a key property that any slice discovery
method should satisfy. We then use influence functions to derive a new slice
discovery method, InfEmbed, which satisfies coherence by returning slices whose
examples are influenced similarly by the training data. InfEmbed is simple, and
consists of applying K-Means clustering to a novel representation we deem
influence embeddings. We show InfEmbed outperforms current state-of-the-art
methods on 2 benchmarks, and is effective for model debugging across several
case studies.