KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inference
This addresses memory constraints for on-device AI applications like meeting summarization, though it is incremental as it builds on existing KV cache offloading schemes.
The paper tackles the memory capacity wall in on-device long-context inference by offloading the KV cache to disk, resulting in higher throughput under tight memory budgets while maintaining generation quality across models and storage types.
Language models (LMs) underpin emerging mobile and embedded AI applications like meeting and video summarization and document analysis, which often require processing multiple long-context inputs. Running an LM locally on-device improves privacy, enables offline use, and reduces cost, but long-context inference quickly hits a \emph{memory capacity wall} as the key-value (KV) cache grows linearly with context length and batch size. We present KVSwap, a software framework to break this memory wall by offloading the KV cache to non-volatile secondary storage (disk). KVSwap leverages the observation that only a small, dynamically changing subset of KV entries is critical for generation. It stores the full cache on disk, uses a compact in-memory metadata to predict which entries to preload, overlaps computation with hardware-aware disk access, and orchestrates read patterns to match storage device characteristics. Our evaluation shows that across representative LMs and storage types, KVSwap delivers higher throughput under tight memory budgets while maintaining the generation quality when compared with existing KV cache offloading schemes.