Is One Score Enough? Rethinking the Evaluation of Sequentially Evolving LLM Memory
For researchers and developers of LLMs with external memory, this work provides a more nuanced evaluation framework that exposes critical memory failures hidden by aggregate metrics.
The paper introduces SeqMem-Eval, a diagnostic framework for evaluating LLM memory that goes beyond aggregate metrics to measure forgetting, negative transfer, and other failure modes. Experiments show that high final accuracy often masks substantial forgetting or negative transfer, revealing trade-offs invisible under standard metrics.
Memory plays a central role in enabling large language models (LLMs) to operate over sequential tasks by accumulating and reusing experience over time. However, existing evaluations of LLM memory mostly rely on aggregate metrics such as final hold-out accuracy or cumulative online performance, which can obscure critical failure modes such as forgetting and negative transfer. In this paper, we introduce SeqMem-Eval, a diagnostic evaluation framework for sequentially evolving LLM memory. Drawing inspiration from continual learning, it targets a test-time setting in which memory is external, prompt-mediated, and updated without modifying model parameters. Rather than focusing only on final performance, SeqMem-Eval evaluates how memory states evolve, generalize, consolidate experience, and retain useful information during sequential inference. Specifically, it measures online utility, hold-out generalization, backward transfer, and forgetting, providing a finer-grained view of memory quality. Through extensive experiments across diverse tasks and memory methods, we show that higher final or cumulative accuracy does not necessarily imply better memory quality: many methods exhibit strong performance gains while suffering from substantial forgetting or negative transfer. Moreover, different memory designs exhibit distinct trade-offs between adaptability and stability that remain invisible under standard evaluation metrics.