PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding
This work addresses the bottleneck of hardware underutilization in LLM inference for users needing faster and scalable decoding on multi-device systems, representing a novel method rather than an incremental improvement.
PipeSpec tackles the problem of sequential stage dependencies in speculative decoding for LLM inference by introducing a hierarchical pipeline framework, achieving up to 2.54x speedup and outperforming state-of-the-art methods in tasks like text summarization and code generation.
Speculative decoding accelerates large language model inference by using smaller draft models to generate candidate tokens for parallel verification. However, current approaches are limited by sequential stage dependencies that prevent full hardware utilization. We present PipeSpec, a framework that generalizes speculative decoding to $k$ models arranged in a hierarchical pipeline, enabling asynchronous execution with lightweight coordination for prediction verification and rollback. Our analytical model characterizes token generation rates across pipeline stages and proves guaranteed throughput improvements over traditional decoding for any non-zero acceptance rate. We further derive closed-form expressions for steady-state verification probabilities that explain the empirical benefits of pipeline depth. Experimental results show that PipeSpec achieves up to 2.54$\times$ speedup while outperforming state-of-the-art methods. We validate PipeSpec across text summarization and code generation tasks using LLaMA 2 and 3 models, demonstrating that pipeline efficiency increases with model depth, providing a scalable approach to accelerating LLM inference on multi-device systems.