Structure Your Data: Towards Semantic Graph Counterfactuals
Angeliki Dimitriou Maria Lymperaiou Giorgos Filandrianos Konstantinos Thomas Giorgos Stamou
Counterfactual explanations (CEs) based on concepts are explanations that
consider alternative scenarios to understand which high-level semantic features
contributed to particular model predictions. In this work, we propose CEs based
on the semantic graphs accompanying input data to achieve more descriptive,
accurate, and human-aligned explanations. Building upon state-of-the-art (SoTA)
conceptual attempts, we adopt a model-agnostic edit-based approach and
introduce leveraging GNNs for efficient Graph Edit Distance (GED) computation.
With a focus on the visual domain, we represent images as scene graphs and
obtain their GNN embeddings to bypass solving the NP-hard graph similarity
problem for all input pairs, an integral part of the CE computation process. We
apply our method to benchmark and real-world datasets with varying difficulty
and availability of semantic annotations. Testing on diverse classifiers, we
find that our CEs outperform previous SoTA explanation models based on
semantics, including both white and black-box as well as conceptual and
pixel-level approaches. Their superiority is proven quantitatively and
qualitatively, as validated by human subjects, highlighting the significance of
leveraging semantic edges in the presence of intricate relationships. Our
model-agnostic graph-based approach is widely applicable and easily extensible,
producing actionable explanations across different contexts.