From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents
This addresses the need for better long-term memory evaluation in personalized agents, though it is incremental as it builds on existing benchmarks.
The authors tackled the problem of evaluating long-term memory in personalized agents by introducing Memora, a benchmark spanning weeks to months of user conversations, which revealed frequent reuse of invalid memories and marginal improvements from memory agents.
Personalized agents that interact with users over long periods must maintain persistent memory across sessions and update it as circumstances change. However, existing benchmarks predominantly frame long-term memory evaluation as fact retrieval from past conversations, providing limited insight into agents' ability to consolidate memory over time or handle frequent knowledge updates. We introduce Memora, a long-term memory benchmark spanning weeks to months long user conversations. The benchmark evaluates three memory-grounded tasks: remembering, reasoning, and recommending. To ensure data quality, we employ automated memory-grounding checks and human evaluation. We further introduce Forgetting-Aware Memory Accuracy (FAMA), a metric that penalizes reliance on obsolete or invalidated memory when evaluating long-term memory. Evaluations of four LLMs and six memory agents reveal frequent reuse of invalid memories and failures to reconcile evolving memories. Memory agents offer marginal improvements, exposing shortcomings in long-term memory for personalized agents.