NextMem: Towards Latent Factual Memory for LLM-based Agents
This addresses memory limitations in LLM-based agents, which is an incremental improvement over existing methods.
The paper tackles the problem of constructing factual memory for LLM-based agents by introducing NextMem, a latent factual memory framework that uses an autoregressive autoencoder and achieves superior performance in retrieval, robustness, and extensibility.
Memory is critical for LLM-based agents to preserve past observations for future decision-making, where factual memory serves as its foundational part. However, existing approaches to constructing factual memory face several limitations. Textual methods impose heavy context and indexing burdens, while parametric methods suffer from catastrophic forgetting and high costs. To address these challenges, we introduce NextMem, a latent factual memory framework that utilizes an autoregressive autoencoder to efficiently construct latent memory while ensuring accurate reconstruction. For better optimization, we propose a two-stage training process, including autoregressive reconstruction alignment and progressive latent substitution. We also incorporate quantization to reduce storage overhead. Extensive experiments demonstrate that NextMem achieves superior performance, and excels in retrieval, robustness, and extensibility properties. We release our code and model checkpoints at https://github.com/nuster1128/NextMem.