VoxZip: Semantic-Anchored Temporal KV Cache Compression for Long-Context Audio Inference
This work addresses the critical memory demands of KV caches for long-context audio inference in Speech LLMs, which is a significant bottleneck for researchers and practitioners working with these models.
This paper introduces VoxZip, a train-free, two-stage framework for compressing KV caches in Speech Large Language Models to address memory bottlenecks during long-context audio inference. It achieves over 90% of uncompressed baseline performance with 20x KV cache compression in long-context scenarios, and at 4x compression, it increases inference throughput by 1.9x and reduces peak memory overhead by 3.3x.
Recent advancements in Speech Large Language Models have demonstrated remarkable capabilities in understanding complex audio tasks. Despite this progress, their long-context inference remains severely bottlenecked by prohibitive KV cache memory demands. Existing text-centric compression methods struggle here, often disrupting speech continuity or discarding crucial semantic cues. To address this, we propose VoxZip, a train-free, two-stage semantic-anchored KV cache compression framework. The first stage uses automatic speech recognition (ASR) transcriptions as explicit semantic anchors to temporally align, compress, and fuse audio tokens, significantly reducing the initial KV cache while elevating token information density. To further improve the compression ratio, the second stage employs a dynamic filtering strategy based on temporally decayed accumulated attention to evict non-essential tokens while mitigating early-token bias. Comprehensive evaluations on Qwen3-Omni across six diverse audio benchmarks demonstrate the superiority of our approach. VoxZip excels in long-audio reasoning and consistently maintains high-fidelity perception on short-form tasks. Notably, it sustains over 90\% of the uncompressed baseline performance even under an aggressive 20x KV cache compression in long-context scenarios. Furthermore, at a 4x compression ratio, VoxZip yields a 1.9x increase in inference throughput alongside a 3.3x reduction in peak memory overhead. Code and models will be available at https://github.com/MM-Speech/VoxZip.