Validating Causal Abstraction Metrics on Simulated Complex Systems
This work addresses the lack of consensus on measuring validity of high-level causal explanations in complex systems, providing a validated metric for scientists and AI researchers.
The paper introduces a benchmark of ten complex systems with ground-truth causal explanations and evaluates over thirty metrics for validating causal abstractions. Only causal metrics with faithfulness testing reliably discriminate valid from invalid abstractions, leading to the Causal Abstraction Error (CAE), which passes all tests and converges with as few as 30 sampled interventions.
A central goal of science is to produce valid explanations of complex systems: high-level causal accounts that faithfully reflect the behavior of lower-level mechanisms. Yet no consensus exists on how to measure whether a proposed high-level explanation is actually valid. We introduce a benchmark of ten complex systems spanning both discrete and continuous state spaces, as well as static and dynamical regimes, each equipped with consensual ground-truth causal explanations and invalid contrastive conditions. Within a unified causal abstraction framework, we systematically evaluate over thirty candidate metrics drawn from observational, functional, information-theoretic, and causal families. Our results show that only the latter reliably discriminates valid from invalid abstractions, and only when incorporating faithfulness testing over unmapped variables. Building on these findings, we introduce the Causal Abstraction Error (CAE), a continuous validity metric with an explicit faithfulness test, which passes all discrimination tests across every system and can converge with as few as 30 sampled interventions. We offer it as a general-purpose metric for the discovery and validation of high-level explanations.