RobustCLEVR: A Benchmark and Framework for Evaluating Robustness in
  Object-centric Learning
Nathan Drenkow Mathias Unberath
Object-centric representation learning offers the potential to overcome
limitations of image-level representations by explicitly parsing image scenes
into their constituent components. While image-level representations typically
lack robustness to natural image corruptions, the robustness of object-centric
methods remains largely untested. To address this gap, we present the
RobustCLEVR benchmark dataset and evaluation framework. Our framework takes a
novel approach to evaluating robustness by enabling the specification of causal
dependencies in the image generation process grounded in expert knowledge and
capable of producing a wide range of image corruptions unattainable in existing
robustness evaluations. Using our framework, we define several causal models of
the image corruption process which explicitly encode assumptions about the
causal relationships and distributions of each corruption type. We generate
dataset variants for each causal model on which we evaluate state-of-the-art
object-centric methods. Overall, we find that object-centric methods are not
inherently robust to image corruptions. Our causal evaluation approach exposes
model sensitivities not observed using conventional evaluation processes,
yielding greater insight into robustness differences across algorithms. Lastly,
while conventional robustness evaluations view corruptions as
out-of-distribution, we use our causal framework to show that even training on
in-distribution image corruptions does not guarantee increased model
robustness. This work provides a step towards more concrete and substantiated
understanding of model performance and deterioration under complex corruption
processes of the real-world.