Causal Conclusions that Flip Repeatedly and Their Justification
This addresses a fundamental limitation in causal inference for researchers, showing that reliable conclusions are inherently unstable, which is incremental as it extends prior unreliability results.
The paper proves that consistent methods for causal discovery from observational data can produce conclusions that flip repeatedly with increasing sample size, even for linear Gaussian or discrete Bayes networks, and argues that the best methods minimize such retractions.
Over the past two decades, several consistent procedures have been designed to infer causal conclusions from observational data. We prove that if the true causal network might be an arbitrary, linear Gaussian network or a discrete Bayes network, then every unambiguous causal conclusion produced by a consistent method from non-experimental data is subject to reversal as the sample size increases any finite number of times. That result, called the causal flipping theorem, extends prior results to the effect that causal discovery cannot be reliable on a given sample size. We argue that since repeated flipping of causal conclusions is unavoidable in principle for consistent methods, the best possible discovery methods are consistent methods that retract their earlier conclusions no more than necessary. A series of simulations of various methods across a wide range of sample sizes illustrates concretely both the theorem and the principle of comparing methods in terms of retractions.