ViT-ReciproCAM: Gradient and Attention-Free Visual Explanations for
  Vision Transformer
Seok-Yong Byun Wonju Lee
This paper presents a novel approach to address the challenges of
understanding the prediction process and debugging prediction errors in Vision
Transformers (ViT), which have demonstrated superior performance in various
computer vision tasks such as image classification and object detection. While
several visual explainability techniques, such as CAM, Grad-CAM, Score-CAM, and
Recipro-CAM, have been extensively researched for Convolutional Neural Networks
(CNNs), limited research has been conducted on ViT. Current state-of-the-art
solutions for ViT rely on class agnostic Attention-Rollout and Relevance
techniques. In this work, we propose a new gradient-free visual explanation
method for ViT, called ViT-ReciproCAM, which does not require attention matrix
and gradient information. ViT-ReciproCAM utilizes token masking and generated
new layer outputs from the target layer's input to exploit the correlation
between activated tokens and network predictions for target classes. Our
proposed method outperforms the state-of-the-art Relevance method in the
Average Drop-Coherence-Complexity (ADCC) metric by $4.58\%$ to $5.80\%$ and
generates more localized saliency maps. Our experiments demonstrate the
effectiveness of ViT-ReciproCAM and showcase its potential for understanding
and debugging ViT models. Our proposed method provides an efficient and
easy-to-implement alternative for generating visual explanations, without
requiring attention and gradient information, which can be beneficial for
various applications in the field of computer vision.