Evaluating Strategic Reasoning in Forecasting Agents
For AI researchers and practitioners, this benchmark provides a controlled environment to evaluate and improve strategic reasoning in forecasting agents, revealing specific weaknesses in current models.
The paper introduces BTF-2, a benchmark of 1,417 questions with a frozen research corpus, enabling reproducible evaluation of forecasting agents. It finds that the best forecaster outperforms any single frontier agent by 0.011 Brier score, and identifies key strategic reasoning failures in frontier agents, such as misjudging incentives and institutional processes.
Forecasting benchmarks produce accuracy leaderboards but little insight into why some forecasters are more accurate than others. We introduce Bench to the Future 2 (BTF-2), 1,417 pastcasting questions with a frozen 15M-document research corpus in which agents reproducibly research and forecast offline, producing full reasoning traces. BTF-2 detects accuracy differences of 0.004 Brier score, and can distinguish differential agent strengths in research vs. judgment. We build a forecaster 0.011 Brier more accurate than any single frontier agent, and use it to evaluate agent strategic reasoning without hindsight bias. We find the better forecaster differs primarily in its pre-mortem analysis of its blind spots and consideration of black swans. Expert human forecasters found the dominant strategic reasoning failures of frontier agents are in assessing political and business leaders' incentives, judging their likelihood to follow through on stated plans, and modeling institutional processes.