SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment
This provides a rigorous evaluation framework for researchers developing self-evolving AI agents, though it's incremental as it builds on the existing SEA paradigm.
The paper tackles the problem that current LLM-based agents lack the ability to accumulate experience across tasks, and introduces SEA-Eval, the first benchmark to evaluate self-evolving agents by measuring success rates and token consumption over sequential tasks, revealing up to 31.2 times differences in token efficiency despite identical success rates.
Current LLM-based agents demonstrate strong performance in episodic task execution but remain constrained by static toolsets and episodic amnesia, failing to accumulate experience or optimize strategies across task boundaries. While the Self-Evolving Agent (SEA) paradigm has been previously proposed, this paper contributes a new formal definition of SEA grounded in digital embodiment and continuous cross-task evolution, and introduces SEA-Eval, the first benchmark designed to evaluate SEA characteristics across two dimensions, intra-task execution reliability and long-term evolutionary performance. By organizing tasks into sequential streams and analyzing Success Rate and Token Consumption over time, SEA-Eval quantifies evolutionary gain and structural stability in ways that existing episodic benchmarks cannot. Empirical evaluations reveal a significant evolutionary bottleneck in current state-of-the-art frameworks, where identical success rates mask up to 31.2 times differences in token consumption and divergent evolutionary trajectories under sequential analysis. SEA-Eval provides a rigorous scientific foundation for advancing agents from mere task executors toward genuinely self-evolving digital entities.