Selective KV Cache Protection for Noise-Resilient LLM Inference on Analog Compute-In-Memory Systems
For researchers and engineers building energy-efficient LLM inference hardware, this work provides a practical solution to a previously unexplored noise problem in analog CIM attention, enabling more reliable deployment.
The paper studies the impact of hardware noise on KV cache operations in analog compute-in-memory systems for LLM inference, and proposes a hierarchical token protection strategy that keeps vulnerable tokens on a digital path. This reduces average perplexity from 33.91 to 11.95 (close to the clean baseline of 11.06) and improves programming-row utilization from 23.1% to 91.2%.
Analog compute-in-memory (CIM) arrays have emerged as a promising substrate for energy-efficient LLM inference, particularly for weight-stationary computations in linear layers. However, extending analog CIM to attention mechanisms introduces a fundamental challenge: KV cache operations demand repeated in-situ weight updates, and the resulting mismatch with the weight-stationary paradigm exposes dynamic computations to significant hardware noise, a critical problem that remains largely unexplored. In this paper, we present the first systematic study of dynamic attention computation on analog CIM arrays, revealing that initial and recent tokens exhibit disproportionate vulnerability to hardware noise. Motivated by this token-level insight, we propose a hierarchical token protection strategy that keeps sink tokens and a sliding recent-token window on a higher-precision digital path while processing the bulk KV cache on analog CIM. A co-designed scheduler combines analog programming, ownership transition, and bulk-MVM tile formation to bound digital overhead. Evaluations on nine LLMs show that our approach lowers average perplexity under analog noise from 33.91 to 11.95, close to the clean baseline of 11.06, while improving dynamic-KV programming-row utilization from 23.1\% to 91.2\%.