You Need an Encoder for Native Position-Independent Caching
This addresses a bottleneck in LLM inference for applications requiring arbitrary context retrieval, offering a practical improvement over existing PIC methods.
The paper tackles the inefficiency of prefix-based KV caching in LLMs by proposing native Position-Independent Caching (PIC) with an encoder, reducing Time-to-First-Token by 51-94% and increasing throughput by 3x while maintaining comparable accuracy.
The Key-Value (KV) cache of Large Language Models (LLMs) is prefix-based, making it highly inefficient for processing contexts retrieved in arbitrary order. Position-Independent Caching (PIC) has been proposed to enable KV reuse without positional constraints; however, existing approaches often incur substantial accuracy degradation, limiting their practical adoption. To address this issue, we propose native PIC by reintroducing the encoder to prevalent decoder-only LLMs and explicitly training it to support PIC. We further develop COMB, a PIC-aware caching system that integrates seamlessly with existing inference frameworks. Experimental results show that COMB reduces Time-to-First-Token (TTFT) by 51-94% and increases throughput by 3$\times$ with comparable accuracy. Furthermore, the quality improvement when using DeepSeek-V2-Lite-Chat demonstrates the applicability of COMB to other types of decoder-only LLMs. Our code is available at https://github.com/shijuzhao/Comb.