Towards Dynamic Causal Discovery with Rare Events: A Nonparametric Conditional Independence Test
This addresses a critical gap for engineers and analysts in fields like safety and risk management, where rare events are consequential but hard to model, though it is incremental as it builds on existing causal discovery frameworks.
The paper tackled the problem of causal discovery in dynamic systems with rare events, where existing methods fail to detect causal links that only appear during low-probability occurrences, and introduced a nonparametric conditional independence test that achieved validated performance on simulated and real-world datasets like Caltrans PeMS incident data.
Causal phenomena associated with rare events occur across a wide range of engineering problems, such as risk-sensitive safety analysis, accident analysis and prevention, and extreme value theory. However, current methods for causal discovery are often unable to uncover causal links, between random variables in a dynamic setting, that manifest only when the variables first experience low-probability realizations. To address this issue, we introduce a novel statistical independence test on data collected from time-invariant dynamical systems in which rare but consequential events occur. In particular, we exploit the time-invariance of the underlying data to construct a superimposed dataset of the system state before rare events happen at different timesteps. We then design a conditional independence test on the reorganized data. We provide non-asymptotic sample complexity bounds for the consistency of our method, and validate its performance across various simulated and real-world datasets, including incident data collected from the Caltrans Performance Measurement System (PeMS). Code containing the datasets and experiments is publicly available.