AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations
This work addresses the challenge of reliable and scalable memory benchmarking for conversational AI developers, though it is incremental as it builds on existing memory approaches by providing a new evaluation framework.
The authors tackled the problem of evaluating and optimizing memory management in LLM-based assistants for long-horizon conversations by introducing AMemGym, an interactive environment that uses structured data sampling and LLM-simulated users to generate high-quality interactions, revealing performance gaps in existing memory systems like RAG and long-context LLMs.
Long-horizon interactions between users and LLM-based assistants necessitate effective memory management, yet current approaches face challenges in training and evaluation of memory. Existing memory benchmarks rely on static, off-policy data as context, limiting evaluation reliability and scalability. To address these gaps, we introduce AMemGym, an interactive environment enabling on-policy evaluation and optimization for memory-driven personalization. AMemGym employs structured data sampling to predefine user profiles, state-dependent questions, and state evolution trajectories, enabling cost-effective generation of high-quality, evaluation-aligned interactions. LLM-simulated users expose latent states through role-play while maintaining structured state consistency. Comprehensive metrics based on structured data guide both assessment and optimization of assistants. Extensive experiments reveal performance gaps in existing memory systems (e.g., RAG, long-context LLMs, and agentic memory) and corresponding reasons. AMemGym not only enables effective selection among competing approaches but also can potentially drive the self-evolution of memory management strategies. By bridging structured state evolution with free-form interactions, our framework provides a scalable, diagnostically rich environment for advancing memory capabilities in conversational agents.