Stable but Wrong: When More Data Degrades Scientific Conclusions
This reveals an intrinsic limit for data-driven science, indicating that stability and confidence are insufficient for validity, which is a foundational issue affecting all fields relying on observational data.
The paper tackles the problem that accumulating more data can degrade scientific conclusions, showing that standard inference procedures can converge smoothly to incorrect results while passing diagnostic checks, with additional data amplifying error rather than correcting it.
Modern science increasingly relies on ever-growing observational datasets and automated inference pipelines, under the implicit belief that accumulating more data makes scientific conclusions more reliable. Here we show that this belief can fail in a fundamental and irreversible way. We identify a structural regime in which standard inference procedures converge smoothly, remain well calibrated, and pass conventional diagnostic checks, yet systematically converge to incorrect conclusions. This failure arises when the reliability of observations degrades in a manner that is intrinsically unobservable to the inference process itself. Using minimal synthetic experiments, we demonstrate that in this regime additional data do not correct error but instead amplify it, while residual-based and goodness-of-fit diagnostics remain misleadingly normal. These results reveal an intrinsic limit of data-driven science: stability, convergence, and confidence are not sufficient indicators of epistemic validity. We argue that inference cannot be treated as an unconditional consequence of data availability, but must instead be governed by explicit constraints on the integrity of the observational process.