SSSD: Simply-Scalable Speculative Decoding
This work addresses deployment inefficiencies in speculative decoding for data center-scale LLM inference, offering a practical solution with significant performance gains.
The paper tackles the challenge of scaling speculative decoding for large language model inference at typical data center batch sizes, achieving a 4x throughput increase with no latency impact for short contexts and 1.7-2x improvements in latency and throughput for longer contexts.
Over the past year, Speculative Decoding has gained popularity as a technique for accelerating Large Language Model inference. While several methods have been introduced, most struggle to deliver satisfactory performance at batch sizes typical for data centers ($\geq 8$) and often involve significant deployment complexities. In this work, we offer a theoretical explanation of how Speculative Decoding can be effectively utilized with larger batch sizes. We also introduce a method that integrates seamlessly into existing systems without additional training or the complexity of deploying a small LLM. In a continuous batching setting, we achieve a 4x increase in throughput without any latency impact for short context generation, and a 1.7-2x improvement in both latency and throughput for longer contexts.