CHESS: Context-aware Hierarchical Efficient Semantic Selection for Long-Context LLM Inference
This addresses efficiency bottlenecks for users of long-context LLMs, offering significant speedups with minimal quality loss, though it is incremental as it builds on prior pruning methods.
The paper tackles the problem of high latency in long-context LLM inference due to KV cache constraints by proposing CHESS, a context-aware hierarchical selection system that achieves comparable quality to full KV cache using only 1% of it and increases throughput by up to 4.56×.
Long-context LLMs demand accurate inference at low latency, yet decoding becomes primarily constrained by KV cache as context grows. Prior pruning methods are largely context-agnostic: their token selection ignores step-wise relevance and local semantics, which undermines quality. Moreover, their irregular accesses and selection overheads yield only limited wall-clock speedups. To address this, we propose \textbf{CHESS}, an \textit{algorithm-system co-design} KV-cache management system. Algorithmically, CHESS introduces a context-aware, hierarchical selection policy that dynamically reconstructs a coherent context for the current decoding. System-wise, coarse granularity selection eliminates expensive data movement, fully realizing practical acceleration from theoretical sparsity. Extensive evaluations demonstrate that CHESS surpasses Full-KV quality using only \textbf{1\%} of the KV cache, delivers low-latency stable inference with up to \textbf{4.56$\times$} higher throughput, and consistently outperforms other strong baselines. Code is available at \href{https://anonymous.4open.science/r/CHESS-9958/}{https://anonymous.4open.science/r/CHESS/}.