SpecPV: Improving Self-Speculative Decoding for Long-Context Generation via Partial Verification
This addresses the problem of slow generation for long-context tasks like code generation and document understanding, offering an incremental improvement to speculative decoding.
The paper tackles the bottleneck of verification in speculative decoding for long-context generation by introducing SpecPV, which uses partial key-value states for fast verification and periodic full verification, achieving up to 6x decoding speedup with minor degradation.
Growing demands from tasks like code generation, deep reasoning, and long-document understanding have made long-context generation a crucial capability for large language models (LLMs). Speculative decoding is one of the most direct and effective approaches for accelerating generation. It follows a draft-verify paradigm, where a lightweight draft model proposes several candidate tokens and the target model verifies them. However, we find that as the context length grows, verification becomes the dominant bottleneck. To further accelerate speculative decoding in long-context generation, we introduce SpecPV, a self-speculative decoding approach that performs fast verification using partial key-value states (KV) and periodically applies full verification to eliminate accumulated errors. We validate SpecPV across multiple long-context benchmarks and models, including LLaMA-3.1-8B-Instruct and Qwen3-series. Experimental results show that SpecPV achieves up to 6x decoding speedup over standard autoregressive decoding with minor degradation.