Generalizing Backpropagation for Gradient-Based Interpretability
Kevin Du Lucas Torroba Hennigen Niklas Stoehr Alexander Warstadt Ryan Cotterell
Many popular feature-attribution methods for interpreting deep neural
networks rely on computing the gradients of a model's output with respect to
its inputs. While these methods can indicate which input features may be
important for the model's prediction, they reveal little about the inner
workings of the model itself. In this paper, we observe that the gradient
computation of a model is a special case of a more general formulation using
semirings. This observation allows us to generalize the backpropagation
algorithm to efficiently compute other interpretable statistics about the
gradient graph of a neural network, such as the highest-weighted path and
entropy. We implement this generalized algorithm, evaluate it on synthetic
datasets to better understand the statistics it computes, and apply it to study
BERT's behavior on the subject-verb number agreement task (SVA). With this
method, we (a) validate that the amount of gradient flow through a component of
a model reflects its importance to a prediction and (b) for SVA, identify which
pathways of the self-attention mechanism are most important.