Assessing Large Language Models in Updating Their Forecasts with New Information
This addresses the need for more robust belief updating in LLMs for dynamic forecasting tasks, though it is incremental as it builds on prior static prediction work.
The paper tackled the problem of evaluating whether large language models appropriately update their forecasts when presented with new information, finding that while LLMs show some responsiveness, their updates are often inconsistent or overly conservative and lag behind human standards.
Prior work has largely treated future event prediction as a static task, failing to consider how forecasts and the confidence in them should evolve as new evidence emerges. To address this gap, we introduce EVOLVECAST, a framework for evaluating whether large language models appropriately revise their predictions in response to new information. In particular, EVOLVECAST assesses whether LLMs adjust their forecasts when presented with information released after their training cutoff. We use human forecasters as a comparative reference to analyze prediction shifts and confidence calibration under updated contexts. While LLMs demonstrate some responsiveness to new information, their updates are often inconsistent or overly conservative. We further find that neither verbalized nor logits-based confidence estimates consistently outperform the other, and both remain far from the human reference standard. Across settings, models tend to express conservative bias, underscoring the need for more robust approaches to belief updating.