Do Real-World Datasets Contain Natural Experiments? An Empirical Study Using Causal Feature Selection
For researchers in causal inference and machine learning, this work provides an initial exploration into detecting and leveraging natural experiments in existing datasets, though it is preliminary and limited in scope.
This paper investigates whether real-world datasets contain natural experiments and proposes a method to detect them using causal feature selection. The authors find that treating data as interventional improves model performance, suggesting natural experiments exist in real-world datasets.
In nature, events that affect some individuals or groups but not others constitute an implicit intervention and are known as natural experiments. For example, the COVID-19 pandemic was an intervention by the coronavirus on the sub-population infected with COVID. We ask, do natural experiments occur in existing real-world datasets? If yes, how should we treat them? To detect natural experiments in data, we use causal discovery to recover the underlying causal graph and perform feature selection based on causal links. If downstream performance improves by treating the data as interventional rather than observational, we argue that this suggests the dataset contains natural experiments. We first validate this hypothesis by simulating datasets with and without natural experiments using synthetic graphs. We then perform a systematic empirical evaluation on a large suite of real-world datasets. Our results indicate that real-world datasets do contain natural experiments and we can take advantage of those natural experiments to improve model performance using causal inference. Our work represents the initial foray into this area, offering a preliminary exploration within a limited scope.