Locomo-Plus: Beyond-Factual Cognitive Memory Evaluation Framework for LLM Agents
This addresses the need for better evaluation of cognitive memory in conversational AI, though it is incremental as it builds on existing memory benchmarks.
The authors tackled the problem of evaluating long-term cognitive memory in LLM-based dialogue systems, where models must retain and apply implicit constraints like user goals, and introduced the LoCoMo-Plus benchmark, showing that current methods fail to capture these challenges.
Long-term conversational memory is a core capability for LLM-based dialogue systems, yet existing benchmarks and evaluation protocols primarily focus on surface-level factual recall. In realistic interactions, appropriate responses often depend on implicit constraints such as user state, goals, or values that are not explicitly queried later. To evaluate this setting, we introduce \textbf{LoCoMo-Plus}, a benchmark for assessing cognitive memory under cue--trigger semantic disconnect, where models must retain and apply latent constraints across long conversational contexts. We further show that conventional string-matching metrics and explicit task-type prompting are misaligned with such scenarios, and propose a unified evaluation framework based on constraint consistency. Experiments across diverse backbone models, retrieval-based methods, and memory systems demonstrate that cognitive memory remains challenging and reveals failures not captured by existing benchmarks. Our code and evaluation framework are publicly available at: https://github.com/xjtuleeyf/Locomo-Plus.