A Practical Investigation of Training-free Relaxed Speculative Decoding
Provides practical guidance for practitioners on when and how to use relaxed speculative decoding to trade off sampling accuracy for speed.
This paper investigates training-free relaxed speculative decoding for LLMs, finding that relaxation requires careful capability evaluation and many methods are unsuitable for lightweight drafters.
Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM's sampling distribution. Recent work argues that relaxing this strict guarantee can yield further speed-ups, controlled capability-speed trade-offs, or even capability gains. We practically investigate training-free relaxed speculative decoding techniques, unify existing approaches within a shared framework, benchmark them on contemporary settings, and distil takeaways and empirical findings for practitioners. Important takeaways include: relaxation can require considerable capability evaluation unlike lossless speculative decoding, and many relaxed approaches rely on a drafter that is a good language model, making them unsuited for lightweight dedicated multi-token-prediction drafters.