Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
This addresses the need for better evaluation methods for agentic search systems, which are increasingly used for web-scale information retrieval, though it is incremental in improving benchmarking rather than introducing a new paradigm.
The paper tackles the problem of evaluating complex agentic search systems by introducing Mind2Web 2, a benchmark of 130 realistic long-horizon web browsing tasks, and proposes an Agent-as-a-Judge framework for automatic assessment, with results showing the best system achieving 50-70% of human performance in half the time.
Agentic search such as Deep Research systems-where agents autonomously browse the web, synthesize information, and return comprehensive citation-backed answers-represents a major shift in how users interact with web-scale information. While promising greater efficiency and cognitive offloading, the growing complexity and open-endedness of agentic search have outpaced existing evaluation benchmarks and methodologies, which largely assume short search horizons and static answers. In this paper, we introduce Mind2Web 2, a benchmark of 130 realistic, high-quality, and long-horizon tasks that require real-time web browsing and extensive information synthesis, constructed with over 1000 hours of human labor. To address the challenge of evaluating time-varying and complex answers, we propose a novel Agent-as-a-Judge framework. Our method constructs task-specific judge agents based on a tree-structured rubric design to automatically assess both answer correctness and source attribution. We conduct a comprehensive evaluation of ten frontier agentic search systems and human performance, along with a detailed error analysis to draw insights for future development. The best-performing system, OpenAI Deep Research, can already achieve 50-70% of human performance while spending half the time, highlighting its great potential. Altogether, Mind2Web 2 provides a rigorous foundation for developing and benchmarking the next generation of agentic search systems.