Over-Searching in Search-Augmented Large Language Models
This addresses computational inefficiency and reliability issues for users of search-augmented LLMs, but it is incremental as it builds on existing retrieval-augmented methods.
The paper tackles the problem of over-searching in search-augmented large language models, where unnecessary searches lead to inefficiency and hallucinations, and finds that over-searching is more pronounced in complex models and noisy retrieval, while introducing a metric called Tokens Per Correctness (TPC) to quantify it.
Search-augmented large language models (LLMs) excel at knowledge-intensive tasks by integrating external retrieval. However, they often over-search -- unnecessarily invoking search tool even when it does not improve response quality, which leads to computational inefficiency and hallucinations by incorporating irrelevant context. In this work, we conduct a systematic evaluation of over-searching across multiple dimensions, including query types, model categories, retrieval conditions, and multi-turn conversations. Our finding shows: (i) search generally improves answer accuracy on answerable queries but harms abstention on unanswerable ones; (ii) over-searching is more pronounced in complex reasoning models and deep research systems, is exacerbated by noisy retrieval, and compounds across turns in multi-turn conversations; and (iii) the composition of retrieved evidence is crucial, as the presence of negative evidence improves abstention. To quantify over-searching, we introduce Tokens Per Correctness (TPC), an evaluation metric that captures the performance-cost trade-off for search-augmented LLMs. Lastly, we investigate mitigation approaches at both the query and retrieval levels and release the OverSearchQA to foster continued research into efficient search-augmented LLMs.