Towards the Visualization of Aggregated Class Activation Maps to Analyse
  the Global Contribution of Class Features
Igor Cherepanov David Sessler Alex Ulmer Hendrik Lücke-Tieke Jörn Kohlhammer
Deep learning (DL) models achieve remarkable performance in classification
tasks. However, models with high complexity can not be used in many
risk-sensitive applications unless a comprehensible explanation is presented.
Explainable artificial intelligence (xAI) focuses on the research to explain
the decision-making of AI systems like DL. We extend a recent method of Class
Activation Maps (CAMs) which visualizes the importance of each feature of a
data sample contributing to the classification. In this paper, we aggregate
CAMs from multiple samples to show a global explanation of the classification
for semantically structured data. The aggregation allows the analyst to make
sophisticated assumptions and analyze them with further drill-down
visualizations. Our visual representation for the global CAM illustrates the
impact of each feature with a square glyph containing two indicators. The color
of the square indicates the classification impact of this feature. The size of
the filled square describes the variability of the impact between single
samples. For interesting features that require further analysis, a detailed
view is necessary that provides the distribution of these values. We propose an
interactive histogram to filter samples and refine the CAM to show relevant
samples only. Our approach allows an analyst to detect important features of
high-dimensional data and derive adjustments to the AI model based on our
global explanation visualization.