Benchmarking Counterfactual Image Generation
Thomas Melistas Nikos Spyrou Nefeli Gkouti Pedro Sanchez Athanasios Vlontzos Giorgos Papanastasiou Sotirios A. Tsaftaris
Counterfactual image generation is pivotal for understanding the causal
relations of variables, with applications in interpretability and generation of
unbiased synthetic data. However, evaluating image generation is a
long-standing challenge in itself. The need to evaluate counterfactual
generation compounds on this challenge, precisely because counterfactuals, by
definition, are hypothetical scenarios without observable ground truths. In
this paper, we present a novel comprehensive framework aimed at benchmarking
counterfactual image generation methods. We incorporate metrics that focus on
evaluating diverse aspects of counterfactuals, such as composition,
effectiveness, minimality of interventions, and image realism. We assess the
performance of three distinct conditional image generation model types, based
on the Structural Causal Model paradigm. Our work is accompanied by a
user-friendly Python package which allows to further evaluate and benchmark
existing and future counterfactual image generation methods. Our framework is
extendable to additional SCM and other causal methods, generative models, and
datasets.