AttnLRP: Attention-Aware Layer-wise Relevance Propagation for
  Transformers
Reduan Achtibat Sayed Mohammad Vakilzadeh Hatefi Maximilian Dreyer Aakriti Jain Thomas Wiegand Sebastian Lapuschkin Wojciech Samek
Large Language Models are prone to biased predictions and hallucinations,
underlining the paramount importance of understanding their model-internal
reasoning process. However, achieving faithful attributions for the entirety of
a black-box transformer model and maintaining computational efficiency is an
unsolved challenge. By extending the Layer-wise Relevance Propagation
attribution method to handle attention layers, we address these challenges
effectively. While partial solutions exist, our method is the first to
faithfully and holistically attribute not only input but also latent
representations of transformer models with the computational efficiency similar
to a singular backward pass. Through extensive evaluations against existing
methods on Llama 2, Flan-T5 and the Vision Transformer architecture, we
demonstrate that our proposed approach surpasses alternative methods in terms
of faithfulness and enables the understanding of latent representations,
opening up the door for concept-based explanations. We provide an open-source
implementation on GitHub https://github.com/rachtibat/LRP-for-Transformers.