The Causal Chambers: Real Physical Systems as a Testbed for AI
  Methodology
Juan L. Gamella Jonas Peters Peter Bühlmann
In some fields of AI, machine learning and statistics, the validation of new
methods and algorithms is often hindered by the scarcity of suitable real-world
datasets. Researchers must often turn to simulated data, which yields limited
information about the applicability of the proposed methods to real problems.
As a step forward, we have constructed two devices that allow us to quickly and
inexpensively produce large datasets from non-trivial but well-understood
physical systems. The devices, which we call causal chambers, are
computer-controlled laboratories that allow us to manipulate and measure an
array of variables from these physical systems, providing a rich testbed for
algorithms from a variety of fields. We illustrate potential applications
through a series of case studies in fields such as causal discovery,
out-of-distribution generalization, change point detection, independent
component analysis, and symbolic regression. For applications to causal
inference, the chambers allow us to carefully perform interventions. We also
provide and empirically validate a causal model of each chamber, which can be
used as ground truth for different tasks. All hardware and software is made
open source, and the datasets are publicly available at causalchamber.org or
through the Python package causalchamber.