WhenLoss: Diagnosing Write and Retrieval Bottlenecks in Long-Context Memory Systems
For practitioners building long-context memory systems, this work identifies that improving the write stage (compression) is more impactful than retrieval, and provides a method to do so.
The paper diagnoses that write-side bottlenecks (compression) dominate retrieval-side bottlenecks in long-context memory systems under fixed budgets, and proposes Expected Predictive Compression (EPC), which uses an LLM at write time to anticipate future questions and preserve minimal supporting evidence. EPC achieves the highest CSM score (0.49 vs. 0.44 for the strongest baseline) across 500 questions with three readers, reducing the write gap to 0.04.
Long-context memory systems often fail under fixed budgets, but end-to-end evaluation does not reveal whether evidence was discarded during compression or preserved but never retrieved. We introduce a four-condition diagnostic protocol that evaluates a fixed reader under truncated full context (TFC), oracle evidence (OE), complete stored memory (CSM), and retrieved memory (RM). Under this fixed-budget LongMemEval setup, write-side gaps exceed retrieval-side gaps for most tested baselines, with four of six baselines robustly write-dominant under our default diagnosis margin. Motivated by this diagnosis, we propose Expected Predictive Compression (EPC), which moves the key decision--what information to retain--to write time by using an LLM to anticipate likely future questions and preserve the minimal supporting evidence under the token budget, while leaving retrieval unchanged at question time. Across all 500 LongMemEval questions with three readers (GPT-5.2, Claude Sonnet 4, Gemini 2.5 Pro), EPC achieves the highest CSM scores among all systems (0.49 vs. 0.44 for Summary (LLM), the strongest baseline), reducing Delta_write to 0.04 while leaving Delta_retr comparable to other LLM-based systems. These results suggest that, on this benchmark and evaluation setup, improving what the write stage preserves is a key avenue for performance gains in the tested systems.