Sequential Integrated Gradients: a simple but effective method for
  explaining language models
Joseph Enguehard
Several explanation methods such as Integrated Gradients (IG) can be
characterised as path-based methods, as they rely on a straight line between
the data and an uninformative baseline. However, when applied to language
models, these methods produce a path for each word of a sentence
simultaneously, which could lead to creating sentences from interpolated words
either having no clear meaning, or having a significantly different meaning
compared to the original sentence. In order to keep the meaning of these
sentences as close as possible to the original one, we propose Sequential
Integrated Gradients (SIG), which computes the importance of each word in a
sentence by keeping fixed every other words, only creating interpolations
between the baseline and the word of interest. Moreover, inspired by the
training procedure of several language models, we also propose to replace the
baseline token "pad" with the trained token "mask". While being a simple
improvement over the original IG method, we show on various models and datasets
that SIG proves to be a very effective method for explaining language models.