LiveMCP-101: Stress Testing and Diagnosing MCP-enabled Agents on Challenging Queries
This work addresses the need for rigorous evaluation of AI agents' tool orchestration capabilities, which is crucial for advancing autonomous systems, though it is incremental in benchmarking methodology.
The authors tackled the problem of benchmarking AI agents' ability to solve multi-step tasks using diverse MCP tools in realistic scenarios, introducing LiveMCP-101, a benchmark of 101 real-world queries, and found that even frontier LLMs achieve a success rate below 60%.
Tool calling has emerged as a critical capability for AI agents to interact with the real world and solve complex tasks. While the Model Context Protocol (MCP) provides a powerful standardized framework for tool integration, there is a significant gap in benchmarking how well AI agents can effectively solve multi-step tasks using diverse MCP tools in realistic, dynamic scenarios. In this work, we present LiveMCP-101, a benchmark of 101 carefully curated real-world queries, refined through iterative LLM rewriting and manual review, that require coordinated use of multiple MCP tools including web search, file operations, mathematical reasoning, and data analysis. Moreover, we introduce a novel evaluation approach that leverages ground-truth execution plans rather than raw API outputs, better reflecting the evolving nature of real-world environments. Experiments show that even frontier LLMs achieve a success rate below 60\%, highlighting major challenges in tool orchestration. Detailed ablations and error analysis further reveal distinct failure modes and inefficiencies in token usage, pointing to concrete directions for advancing current models. LiveMCP-101 sets a rigorous standard for evaluating real-world agent capabilities, advancing toward autonomous AI systems that reliably execute complex tasks through tool use.