Cartridges at Scale: Training Modular KV Caches over Large Document Collections
This work addresses the scalability and compositionality limitations of monolithic KV caches for large document collections, enabling efficient long-context reasoning in LLMs.
Cartridges at Scale (CAS) introduces a training framework for modular KV caches that scales to collections exceeding a million tokens, improving accuracy by 10-31 points over monolithic cartridges while matching or exceeding RAG accuracy with 3-4x fewer prompt tokens.
Large Language Models can reason over long contexts, yet prefilling millions of tokens is wasteful as much of the content remains static across queries. Cartridges address this by distilling document collections into reusable key-value (KV) caches that eliminate prefilling while preserving accuracy. A critical limitation of this approach is that cartridges are monolithic and non-compositional: encoding an entire collection into a single KV block does not scale, and naively mixing cartridges trained in isolation collapses performance to near chance. We introduce Cartridges at Scale (CAS), a training framework for scalable multi-cartridge learning with dynamic distractor mixing and a memory-efficient budget manager that rotates hundreds of per-document cartridges between GPU and persistent storage. Our approach scales to collections exceeding a million tokens, improving over a monolithic cartridge by 10-31 points at comparable token budgets. Oracle cartridge accuracy falls within 2-6 points of full in-context learning even at high compression. When paired with retrieval for cartridge selection, CAS matches or exceeds conventional RAG accuracy while consuming 3-4x fewer prompt tokens.