FastEagle: Cascaded Drafting for Accelerating Speculative Decoding
This work addresses the bottleneck of sequential dependencies in drafting for LLM inference acceleration, offering a practical improvement for efficient text generation.
The paper tackled the problem of slow sequential drafting in speculative decoding for LLM inference by introducing FastEagle, a non-autoregressive cascaded drafter that emits entire drafts in a single forward pass, resulting in substantial wall-clock speedups over strong autoregressive drafters like EAGLE-3 across multiple LLMs and tasks while maintaining competitive acceptance behavior.
Speculative decoding accelerates generation by drafting candidates and verifying them in parallel, yet state-of-the-art drafters (e.g., EAGLE) still require N sequential passes to propose N tokens. We present FastEagle, a non-autoregressive cascaded drafter that emits an entire draft in a single forward pass. FastEagle replaces temporal steps with a lightweight layer cascade and trains with layer-wise supervision to mitigate error accumulation. Coupled with a constrained draft tree that preserves lossless verification cost, FastEagle delivers substantial wall-clock speedups over strong autoregressive drafters while maintaining competitive acceptance behavior. Across multiple LLMs (Vicuna-13B, LLaMA-Instruct 3.x, and DeepSeek-R1-Distill-LLaMA) and tasks (MT-Bench, HumanEval, GSM8K, CNN/DM, Alpaca), FastEagle consistently outperforms EAGLE-3 in speedup under both greedy and stochastic decoding, with comparable average acceptance lengths. These results indicate that removing sequential dependencies in drafting is a practical path toward lossless LLM inference acceleration.