MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks
This addresses a gap in benchmarking for AI agents with memory in realistic, interdependent tasks, though it is incremental as it builds on existing evaluation frameworks.
The authors tackled the problem of evaluating agent memory in isolation by introducing MemoryArena, a benchmark for multi-session tasks where memory and action are interdependent, and found that agents performing well on existing benchmarks like LoCoMo perform poorly in this setting.
Existing evaluations of agents with memory typically assess memorization and action in isolation. One class of benchmarks evaluates memorization by testing recall of past conversations or text but fails to capture how memory is used to guide future decisions. Another class focuses on agents acting in single-session tasks without the need for long-term memory. However, in realistic settings, memorization and action are tightly coupled: agents acquire memory while interacting with the environment, and subsequently rely on that memory to solve future tasks. To capture this setting, we introduce MemoryArena, a unified evaluation gym for benchmarking agent memory in multi-session Memory-Agent-Environment loops. The benchmark consists of human-crafted agentic tasks with explicitly interdependent subtasks, where agents must learn from earlier actions and feedback by distilling experiences into memory, and subsequently use that memory to guide later actions to solve the overall task. MemoryArena supports evaluation across web navigation, preference-constrained planning, progressive information search, and sequential formal reasoning, and reveals that agents with near-saturated performance on existing long-context memory benchmarks like LoCoMo perform poorly in our agentic setting, exposing a gap in current evaluations for agents with memory.