Look Back to Reason Forward: Revisitable Memory for Long-Context LLM Agents
This addresses challenges in long-context reasoning for AI agents, though it is incremental as it builds on existing memory-augmented methods.
The paper tackled the problem of long-context question answering in large language models, where evidence is scattered across millions of tokens, by introducing ReMemR1 with revisitable memory and RLMLR for training, achieving significant gains over existing memory-based approaches.
Large language models face challenges in long-context question answering, where key evidence of a query may be dispersed across millions of tokens. Existing works equip large language models with a memory corpus that is dynamically updated during a single-pass document scan, also known as the "memorize while reading" methods. While this approach scales efficiently, it suffers from irreversible forward-only processing, information loss through overwriting, and sparse reinforcement learning signals. To tackle these challenges, we present ReMemR1, a memory-augmented agent with callback-enhanced memory that allows selective retrieval from the entire memory history and allows non-linear reasoning and revisiting of early evidence. To further strengthen training, we propose Reinforcement Learning with Multi-Level Rewards (RLMLR), which combines final-answer rewards with dense, step-level signals that guide effective memory use. Together, these contributions mitigate information degradation, improve supervision, and support multi-hop memory utilizing. Experiments on long-document QA show significant gains over existing memory-based approaches, which validates ReMemR1 as an effective solution for long-context reasoning agents.