AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts
This addresses the need for better benchmarks to simulate agent-environment interactions in AI, though it is incremental as it builds on existing long-context evaluation methods.
The paper tackles the problem of evaluating autonomous agents in dynamic, long-context scenarios by introducing AgentLongBench, a benchmark based on environment rollouts using Lateral Thinking Puzzles, and finds that agents struggle with dynamic information synthesis, with performance degrading as token requirements increase.
The evolution of Large Language Models (LLMs) into autonomous agents necessitates the management of extensive, dynamic contexts. Current benchmarks, however, remain largely static, relying on passive retrieval tasks that fail to simulate the complexities of agent-environment interaction, such as non-linear reasoning and iterative feedback. To address this, we introduce \textbf{AgentLongBench}, which evaluates agents through simulated environment rollouts based on Lateral Thinking Puzzles. This framework generates rigorous interaction trajectories across knowledge-intensive and knowledge-free scenarios. Experiments with state-of-the-art models and memory systems (32K to 4M tokens) expose a critical weakness: while adept at static retrieval, agents struggle with the dynamic information synthesis essential for workflows. Our analysis indicates that this degradation is driven by the minimum number of tokens required to resolve a query. This factor explains why the high information density inherent in massive tool responses poses a significantly greater challenge than the memory fragmentation typical of long-turn dialogues.