Counterfactual Image Editing
Yushu Pan Elias Bareinboim
Counterfactual image editing is an important task in generative AI, which
asks how an image would look if certain features were different. The current
literature on the topic focuses primarily on changing individual features while
remaining silent about the causal relationships between these features, as
present in the real world. In this paper, we formalize the counterfactual image
editing task using formal language, modeling the causal relationships between
latent generative factors and images through a special type of model called
augmented structural causal models (ASCMs). Second, we show two fundamental
impossibility results: (1) counterfactual editing is impossible from i.i.d.
image samples and their corresponding labels alone; (2) even when the causal
relationships between the latent generative factors and images are available,
no guarantees regarding the output of the model can be provided. Third, we
propose a relaxation for this challenging problem by approximating
non-identifiable counterfactual distributions with a new family of
counterfactual-consistent estimators. This family exhibits the desirable
property of preserving features that the user cares about across both factual
and counterfactual worlds. Finally, we develop an efficient algorithm to
generate counterfactual images by leveraging neural causal models.