Decoding Speculative Decoding
This work addresses the efficiency bottleneck in LLM inference for users needing faster deployment, though it is incremental as it optimizes an existing technique.
The paper investigates factors affecting the performance of speculative decoding for LLMs, finding that draft model latency is more critical than language modeling capability, and designs a new draft model that achieves 111% higher throughput than existing models, generalizing across LLaMA variants and fine-tuned models.
Speculative Decoding is a widely used technique to speed up inference for Large Language Models (LLMs) without sacrificing quality. When performing inference, speculative decoding uses a smaller draft model to generate speculative tokens and then uses the target LLM to verify those draft tokens. The speedup provided by speculative decoding heavily depends on the choice of the draft model. In this work, we perform a detailed study comprising over 350 experiments with LLaMA-65B and OPT-66B using speculative decoding and delineate the factors that affect the performance gain provided by speculative decoding. Our experiments indicate that the performance of speculative decoding depends heavily on the latency of the draft model, and the draft model's capability in language modeling does not correlate strongly with its performance in speculative decoding. Based on these insights we explore a new design space for draft models and design hardware-efficient draft models for speculative decoding. Our newly designed draft model can provide 111% higher throughput than existing draft models and our approach generalizes further to all LLaMA models (1/2/3.1) and supervised fine-tuned models.