DecompX: Explaining Transformers Decisions by Propagating Token
  Decomposition
Ali Modarressi Mohsen Fayyaz Ehsan Aghazadeh Yadollah Yaghoobzadeh Mohammad Taher Pilehvar
An emerging solution for explaining Transformer-based models is to use
vector-based analysis on how the representations are formed. However, providing
a faithful vector-based explanation for a multi-layer model could be
challenging in three aspects: (1) Incorporating all components into the
analysis, (2) Aggregating the layer dynamics to determine the information flow
and mixture throughout the entire model, and (3) Identifying the connection
between the vector-based analysis and the model's predictions. In this paper,
we present DecompX to tackle these challenges. DecompX is based on the
construction of decomposed token representations and their successive
propagation throughout the model without mixing them in between layers.
Additionally, our proposal provides multiple advantages over existing solutions
for its inclusion of all encoder components (especially nonlinear feed-forward
networks) and the classification head. The former allows acquiring precise
vectors while the latter transforms the decomposition into meaningful
prediction-based values, eliminating the need for norm- or summation-based
vector aggregation. According to the standard faithfulness evaluations, DecompX
consistently outperforms existing gradient-based and vector-based approaches on
various datasets. Our code is available at
https://github.com/mohsenfayyaz/DecompX.