Token Transformation Matters: Towards Faithful Post-hoc Explanation for
  Vision Transformer
Junyi Wu Bin Duan Weitai Kang Hao Tang Yan Yan
While Transformers have rapidly gained popularity in various computer vision
applications, post-hoc explanations of their internal mechanisms remain largely
unexplored. Vision Transformers extract visual information by representing
image regions as transformed tokens and integrating them via attention weights.
However, existing post-hoc explanation methods merely consider these attention
weights, neglecting crucial information from the transformed tokens, which
fails to accurately illustrate the rationales behind the models' predictions.
To incorporate the influence of token transformation into interpretation, we
propose TokenTM, a novel post-hoc explanation method that utilizes our
introduced measurement of token transformation effects. Specifically, we
quantify token transformation effects by measuring changes in token lengths and
correlations in their directions pre- and post-transformation. Moreover, we
develop initialization and aggregation rules to integrate both attention
weights and token transformation effects across all layers, capturing holistic
token contributions throughout the model. Experimental results on segmentation
and perturbation tests demonstrate the superiority of our proposed TokenTM
compared to state-of-the-art Vision Transformer explanation methods.