Quantitative probing: Validating causal models using quantitative domain knowledge
This work addresses the validation of causal models for researchers and practitioners, but it appears incremental as an enhancement of existing causal validation strategies.
The authors tackled the problem of validating causal models by introducing quantitative probing, a model-agnostic framework that uses quantitative domain knowledge, analogous to train/test splits in correlation-based ML. They demonstrated its effectiveness through simulations on Pearl's sprinkler example and identified limits to guide future research.
We present quantitative probing as a model-agnostic framework for validating causal models in the presence of quantitative domain knowledge. The method is constructed as an analogue of the train/test split in correlation-based machine learning and as an enhancement of current causal validation strategies that are consistent with the logic of scientific discovery. The effectiveness of the method is illustrated using Pearl's sprinkler example, before a thorough simulation-based investigation is conducted. Limits of the technique are identified by studying exemplary failing scenarios, which are furthermore used to propose a list of topics for future research and improvements of the presented version of quantitative probing. The code for integrating quantitative probing into causal analysis, as well as the code for the presented simulation-based studies of the effectiveness of quantitative probing is provided in two separate open-source Python packages.