LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledge
This addresses the need for systematic, long-term assessment of LLMs under evolving knowledge, shifting evaluation from static memorization to tasks requiring up-to-date retrieval and reasoning, though it is incremental as it builds on existing benchmark methodologies.
The authors tackled the problem of evaluating large language models on dynamic knowledge by introducing LiveSearchBench, an automated pipeline for constructing retrieval-dependent benchmarks from recent knowledge updates, which revealed a pronounced performance drop when models confront facts post-dating pretraining, with gaps most salient on multi-hop queries.
Evaluating large language models (LLMs) on question answering often relies on static benchmarks that reward memorization and understate the role of retrieval, failing to capture the dynamic nature of world knowledge. We present LiveSearchBench, an automated pipeline for constructing retrieval-dependent benchmarks from recent knowledge updates. Our method computes deltas between successive Wikidata snapshots, filters candidate triples for quality, and synthesizes natural-language questions at three levels of reasoning difficulty, each guaranteed to admit a unique, verifiable answer through SPARQL validation. The pipeline is fully automated, scalable across time, and minimizes human intervention, enabling continual regeneration of temporally grounded benchmarks. Experiments show a pronounced performance drop when models confront facts that post-date pretraining, with the gap most salient on multi-hop queries. Retrieval augmented methods and larger, instruction-tuned models provide partial gains but fail to close this recency gap. By design, LiveSearchBench shifts evaluation from static memorization toward tasks that require up-to-date retrieval and reasoning, offering a foundation for systematic, long-term assessment of LLMs under evolving knowledge.