The Oracle's Fingerprint: Correlated AI Forecasting Errors and the Limits of Bias Transmission
For AI safety and collective intelligence researchers, it reveals that current LLMs share failure modes but have not yet transmitted these biases to human forecasters, suggesting a latent risk.
The paper finds that three major LLMs (GPT-4o, Claude, Gemini) exhibit highly correlated forecasting errors (mean r=0.77) on 568 binary questions, indicating an epistemic monoculture. However, human forecasts have not yet been significantly biased by LLMs; post-ChatGPT, human errors became less similar to LLM bias patterns.
When large language models (LLMs) are consulted as forecasting tools, the independence of individual errors -- the foundation of collective intelligence -- may collapse. We test three conditions necessary for this "epistemic monoculture" to emerge. In Study 1, we show that GPT-4o, Claude, and Gemini exhibit highly correlated forecasting errors on 568 resolved binary prediction questions (mean pairwise error correlation r = 0.77, p < 0.001; r = 0.78 excluding likely-leaked questions), despite being developed independently by different organizations. In Study 2, we test whether this correlated bias has propagated into human crowd forecasts, using a within-question design that tracks community prediction shifts across the ChatGPT launch boundary (November 2022). We find that community forecasts move in the direction predicted by LLMs (r = 0.20, p = 0.007), but this shift is fully explained by rational updating toward ground truth. In Study 3, we examine whether the category-level pattern of human forecasting errors increasingly resembles the LLM bias fingerprint. We find the opposite: pre-ChatGPT human biases already strongly resembled the LLM pattern (r = 0.87), while post-ChatGPT the resemblance weakened (r = -0.28). Together, these findings reveal an epistemic monoculture that is built but not yet activated: three nominally independent AI systems share the same failure modes, amplifying precisely the biases humans already hold.