Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation
This addresses a key bottleneck in deploying generative models for real-world applications by significantly accelerating inference, though it is an incremental improvement over existing draft-then-verify methods.
The paper tackles the problem of slow autoregressive decoding in sequence-to-sequence models by introducing Speculative Decoding, which uses a draft-then-verify approach to achieve around 5x speedup on tasks like machine translation and summarization while maintaining comparable quality to beam search.
We propose Speculative Decoding (SpecDec), for the first time ever, to formally study exploiting the idea of speculative execution to accelerate autoregressive (AR) decoding. Speculative Decoding has two innovations: Spec-Drafter -- an independent model specially optimized for efficient and accurate drafting -- and Spec-Verification -- a reliable method for verifying the drafted tokens efficiently in the decoding paradigm. Experimental results on various seq2seq tasks including machine translation and abstractive summarization show our approach can achieve around $5\times$ speedup for the popular Transformer architectures with comparable generation quality to beam search decoding, refreshing the impression that the draft-then-verify paradigm introduces only $1.4\times$$\sim$$2\times$ speedup. In addition to the remarkable speedup, we also demonstrate 3 additional advantages of SpecDec, revealing its practical value for accelerating generative models in real-world applications. Our models and codes are available at https://github.com/hemingkx/SpecDec.