Don't Think Twice! Over-Reasoning Impairs Confidence Calibration
This addresses the issue of robust calibration for LLMs in knowledge-intensive tasks, revealing a critical bottleneck that challenges current paradigms, with implications for AI deployment in domains like human and planetary health.
The study tackled the problem of overconfidence in large language models used for question answering by evaluating how reasoning capabilities and computational budget affect confidence calibration, finding that increasing reasoning budgets impairs calibration with 48.7% accuracy for reasoning models, while search-augmented generation achieves 89.3% accuracy.
Large Language Models deployed as question answering tools require robust calibration to avoid overconfidence. We systematically evaluate how reasoning capabilities and budget affect confidence assessment accuracy, using the ClimateX dataset (Lacombe et al., 2023) and expanding it to human and planetary health. Our key finding challenges the "test-time scaling" paradigm: while recent reasoning LLMs achieve 48.7% accuracy in assessing expert confidence, increasing reasoning budgets consistently impairs rather than improves calibration. Extended reasoning leads to systematic overconfidence that worsens with longer thinking budgets, producing diminishing and negative returns beyond modest computational investments. Conversely, search-augmented generation dramatically outperforms pure reasoning, achieving 89.3% accuracy by retrieving relevant evidence. Our results suggest that information access, rather than reasoning depth or inference budget, may be the critical bottleneck for improved confidence calibration of knowledge-intensive tasks.