Look It Up: Analysing Internal Web Search Capabilities of Modern LLMs
This work addresses the reliability of web search integration in LLMs for users needing real-time factual accuracy, but it is incremental as it builds on existing benchmarks and highlights persistent issues.
The paper tackled the problem of evaluating whether modern large language models efficiently use internal web search when needed, finding that web access improves static accuracy for models like GPT-5-mini and Claude Haiku 4.5 but worsens confidence calibration and leads to below 70% accuracy on dynamic queries due to weak query formulation.
Modern large language models integrate web search to provide real-time answers, yet it remains unclear whether they are efficiently calibrated to use search when it is actually needed. We introduce a benchmark evaluating both the necessity and effectiveness of web access across commercial models with no access to internal states or parameters. The dataset includes a static split of 783 temporally anchored questions answerable from pre-cutoff knowledge, aimed at testing whether models invoke search based on low internal confidence, and a dynamic split of 288 post-cutoff queries designed to test whether models recognise when search is required and retrieve updated information. Web access substantially improves static accuracy for GPT-5-mini and Claude Haiku 4.5, though confidence calibration worsens. On dynamic queries, both models frequently invoke search yet remain below 70 percent accuracy due to weak query formulation. Costs per accuracy-improving call remain low, but returns diminish once initial retrieval fails. Selective invocation helps, but models become overconfident and inconsistent after search. Overall, built-in web search meaningfully improves factual accuracy and can be invoked selectively, yet models remain overconfident, skip retrieval when it is essential, and falter once initial search queries underperform. Taken together, internal web search works better as a good low-latency verification layer than a reliable analytical tool, with clear room for improvement.