SPIRe: Boosting LLM Inference Throughput with Speculative Decoding
This work addresses the cost reduction per token for large-scale LLM deployments, though it is incremental as it builds on existing speculative decoding methods.
The paper tackles the problem of increasing throughput for large-batch LLM inference using speculative decoding, achieving over 100% throughput improvement compared to a smaller draft model and over 35% compared to sparse self-speculation.
Speculative decoding (SD) has been shown to reduce the latency of autoregressive decoding (AD) by 2-3x for small batch sizes. However, increasing throughput and therefore reducing the cost per token requires decoding with large batch sizes. Recent work shows that SD can accelerate decoding with large batch sizes too if the context is sufficiently long and the draft model's KV cache is sparse. We introduce SPIRe, a draft model that combines static sparse attention, pruned initialization, and feedback memory to increase the modeled throughput of speculative decoding by over 100% compared to speculation with a much smaller draft model and by over 35% compared to the strong baseline of sparse self-speculation. Our approach is particularly effective when context lengths vary significantly across requests.