Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
It addresses the problem of slow inference for users of large language models, but is incremental as it synthesizes existing research rather than introducing new methods.
This survey tackles the computational inefficiency of autoregressive decoding in large language models by reviewing speculative decoding methods, which use a two-stage drafting and verification framework to improve inference speed.
Efficient inference in large language models (LLMs) has become a critical focus as their scale and complexity grow. Traditional autoregressive decoding, while effective, suffers from computational inefficiencies due to its sequential token generation process. Speculative decoding addresses this bottleneck by introducing a two-stage framework: drafting and verification. A smaller, efficient model generates a preliminary draft, which is then refined by a larger, more sophisticated model. This paper provides a comprehensive survey of speculative decoding methods, categorizing them into draft-centric and model-centric approaches. We discuss key ideas associated with each method, highlighting their potential for scaling LLM inference. This survey aims to guide future research in optimizing speculative decoding and its integration into real-world LLM applications.