MemShare: Memory Efficient Inference for Large Reasoning Models through KV Cache Reuse
This addresses memory efficiency for users of large reasoning models, though it is incremental as it builds on existing KV cache management methods.
The paper tackles the memory overhead problem in Large Reasoning Models during inference by proposing MemShare, a KV cache management approach that reuses similar intermediate reasoning steps, resulting in up to 84.79% improvement in throughput while maintaining accuracy.
Large Reasoning Models (LRMs) have achieved significant advances in mathematical reasoning and formal logic tasks. However, their tendency to generate lengthy chain-of-thought sequences leads to substantial memory overhead during inference. We observe that LRMs frequently produce highly similar intermediate reasoning steps, which correspond to similar KV cache states across layers. Motivated by this observation, we propose MemShare, a novel KV cache management approach that effectively reduces memory overhead. MemShare employs a collaborative filtering algorithm to efficiently identify reusable KV cache blocks and enables zero copy cache reuse to significantly reduce memory overhead, improve throughput while maintaining accuracy. Experimental results demonstrate that MemShare delivers up to 84.79\% improvement in throughput while maintaining better accuracy compared to existing KV cache management methods.