The Causal Chambers: Real Physical Systems as a Testbed for AI Methodology
This provides a practical testbed for researchers in AI and statistics to evaluate algorithms on real physical data, addressing a common bottleneck in method validation.
The authors tackled the scarcity of real-world datasets for validating AI methods by constructing two physical devices called causal chambers, which generate large datasets from well-understood systems and enable interventions for tasks like causal discovery, with all hardware, software, and datasets made open source.
In some fields of AI, machine learning and statistics, the validation of new methods and algorithms is often hindered by the scarcity of suitable real-world datasets. Researchers must often turn to simulated data, which yields limited information about the applicability of the proposed methods to real problems. As a step forward, we have constructed two devices that allow us to quickly and inexpensively produce large datasets from non-trivial but well-understood physical systems. The devices, which we call causal chambers, are computer-controlled laboratories that allow us to manipulate and measure an array of variables from these physical systems, providing a rich testbed for algorithms from a variety of fields. We illustrate potential applications through a series of case studies in fields such as causal discovery, out-of-distribution generalization, change point detection, independent component analysis, and symbolic regression. For applications to causal inference, the chambers allow us to carefully perform interventions. We also provide and empirically validate a causal model of each chamber, which can be used as ground truth for different tasks. All hardware and software is made open source, and the datasets are publicly available at causalchamber.org or through the Python package causalchamber.