Speculative Decoding: Performance or Illusion?
This work addresses the unclear real-world effectiveness of SD for accelerating LLM inference, highlighting performance limitations and research gaps, though it is incremental as it builds on existing SD variants.
The study systematically evaluated speculative decoding (SD) on a production-grade inference engine, revealing that verification by the target model dominates execution and acceptance length varies significantly, with substantial gaps between observed and theoretical speedup bounds.
Speculative decoding (SD) has become a popular technique to accelerate Large Language Model (LLM) inference, yet its real-world effectiveness remains unclear as prior evaluations rely on research prototypes and unrealistically small batch sizes. We present, to our knowledge, the first systematic study of SD on a production-grade and widely deployed inference engine (vLLM), covering multiple SD variants ($n$-gram, EAGLE/EAGLE-3, Draft-Model, Multi-Token Prediction) across diverse workloads, model scales, and batch sizes. We analyze key factors governing SD performance, and quantify a theoretical upper bound on SD speedup. Our results show that verification by the target model dominates the execution, while acceptance length varies markedly across output token positions, requests, and datasets. Comparing measured performance with theoretical bounds reveals substantial gaps between observed and theoretical upper bounds, and we leverage this observation to highlight new research opportunities that our study opens up in improving SD.