Context Distillation as Latent Memory Management
This work addresses the problem of managing multiple distilled contexts in large language models, enabling more efficient and robust memory retrieval for practitioners.
The paper formulates context distillation as a latent memory management problem, introducing a modular memory bank of LoRA adapters with retrieval, routing, and a Self-Gating mechanism to selectively activate memories. Experiments show substantial outperformance over baselines with retrieval, and Self-Gating improves robustness by deactivating unnecessary latent memories.
Context distillation compresses contextual information into model parameters, yet existing methods often ignore how multiple distilled latent memories should be stored, retrieved, and safely activated in non-oracle settings. We formulate context distillation as a latent memory management problem. We distill each context into an independent LoRA adapter, forming a modular memory bank that enables explicit memory selection. Given a query, our framework retrieves candidate memories, routes the query to the most suitable adapter, and uses a Self-Gating mechanism to decide whether latent memory should be activated. To improve efficiency, we further introduce cache sharing to reduce management overhead during inference. Experiments show that our method substantially outperforms baselines with retrieval, while Self-Gating improves robustness by deactivate unnecessary latent memories.