AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments
For researchers and developers of LLM agents, this benchmark exposes the gap between current agent capabilities and real-world deployment demands, highlighting the need for more realistic evaluation.
Existing benchmarks evaluate LLM agents in idealized settings, underestimating real-world challenges. AgentGym2 introduces a framework with real-world task instances, testing agents on end-to-end procedures, tool discovery, composition, and robustness to noise; experiments show even top models like Gemini and GPT-5 struggle, revealing a large capability gap.
Language agents, i.e., LLM agents, progress rapidly and are increasingly deployed in production environments. This trend underscores the urgent need for rigorous and realistic evaluations. However, most existing benchmarks evaluate agents in simplified, idealized settings. They typically rely on pre-packaged tool interfaces, overlook critical steps, and assume inputs are clean and fully specified. Consequently, they understate the difficulty of real deployments, where uncertainty and noise are ubiquitous and agents must proactively explore the environment to uncover new tools. To bridge this gap, we present AgentGym2, a new evaluation framework with task instances grounded in real-world end-to-end working demands. Beyond reasoning and planning, it measures agents' ability to execute end-to-end procedures, discover tools via exploration, compose tools for unseen tasks, and remain robust to noisy and underspecified information. Experiments on 15 proprietary and open-source models show that even SOTA systems like Gemini and GPT-5 struggle on AgentGym2, revealing a substantial gap between the capability of current agents and the demands of real-world applications.