AdaEAGLE: Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures
This work addresses the need for faster and more efficient LLM inference, representing an incremental improvement over existing speculative decoding methods.
The paper tackles the problem of accelerating Large Language Model inference by improving speculative decoding with adaptive draft structures, achieving a 1.62x speedup over vanilla decoding while maintaining output quality.
Speculative Decoding (SD) is a popular lossless technique for accelerating the inference of Large Language Models (LLMs). We show that the decoding speed of SD frameworks with static draft structures can be significantly improved by incorporating context-aware adaptive draft structures. However, current studies on adaptive draft structures are limited by their performance, modeling approaches, and applicability. In this paper, we introduce AdaEAGLE, the first SD framework that explicitly models adaptive draft structures. AdaEAGLE leverages the Lightweight Draft Length Predictor (LDLP) module to explicitly predict the optimal number of draft tokens during inference to guide the draft model. It achieves comparable speedup results without manual thresholds and allows for deeper, more specialized optimizations. Moreover, together with threshold-based strategies, AdaEAGLE achieves a $1.62\times$ speedup over the vanilla AR decoding and outperforms fixed-length SotA baseline while maintaining output quality.