The Wisdom of Deliberating AI Crowds: Does Deliberation Improve LLM-Based Forecasting?
This addresses the problem of improving forecasting accuracy for AI researchers and practitioners, though it is incremental as it builds on known human deliberation effects.
The study investigated whether allowing LLMs to review each other's forecasts improves accuracy in binary forecasting tasks, finding that it significantly reduced Log Loss by 0.020 (about 4%) for diverse models with shared information, but showed no benefit for homogeneous models.
Structured deliberation has been found to improve the performance of human forecasters. This study investigates whether a similar intervention, i.e. allowing LLMs to review each other's forecasts before updating, can improve accuracy in large language models (GPT-5, Claude Sonnet 4.5, Gemini Pro 2.5). Using 202 resolved binary questions from the Metaculus Q2 2025 AI Forecasting Tournament, accuracy was assessed across four scenarios: (1) diverse models with distributed information, (2) diverse models with shared information, (3) homogeneous models with distributed information, and (4) homogeneous models with shared information. Results show that the intervention significantly improves accuracy in scenario (2), reducing Log Loss by 0.020 or about 4 percent in relative terms (p = 0.017). However, when homogeneous groups (three instances of the same model) engaged in the same process, no benefit was observed. Unexpectedly, providing LLMs with additional contextual information did not improve forecast accuracy, limiting our ability to study information pooling as a mechanism. Our findings suggest that deliberation may be a viable strategy for improving LLM forecasting.