A Theoretical Perspective for Speculative Decoding Algorithm
This work provides theoretical insights for researchers and practitioners in AI/ML to optimize inference efficiency in large language models, though it is incremental as it builds on existing empirical methods.
The paper tackles the lack of theoretical understanding of Speculative Decoding, an algorithm that accelerates large language model inference by using a small model to draft tokens and a large model to validate. It analyzes theoretical limits, batch algorithms, and trade-offs between output quality and inference acceleration, revealing fundamental connections via total variation distances.
Transformer-based autoregressive sampling has been the major bottleneck for slowing down large language model inferences. One effective way to accelerate inference is \emph{Speculative Decoding}, which employs a small model to sample a sequence of draft tokens and a large model to validate. Given its empirical effectiveness, the theoretical understanding of Speculative Decoding is falling behind. This paper tackles this gap by conceptualizing the decoding problem via markov chain abstraction and studying the key properties, \emph{output quality and inference acceleration}, from a theoretical perspective. Our analysis covers the theoretical limits of speculative decoding, batch algorithms, and output quality-inference acceleration tradeoffs. Our results reveal the fundamental connections between different components of LLMs via total variation distances and show how they jointly affect the efficiency of decoding algorithms.