Dynamic Depth Decoding: Faster Speculative Decoding for LLMs
This provides faster inference for LLM users, but it is incremental as it builds directly on EAGLE-2.
The paper tackles the problem of accelerating Large Language Models (LLMs) with speculative decoding by introducing Dynamic Depth Decoding (DDD), which optimizes the state-of-the-art EAGLE-2 method with a dynamic depth approach, achieving an average speedup of 3.16x.
The acceleration of Large Language Models (LLMs) with speculative decoding provides a significant runtime improvement without any loss of accuracy. Currently, EAGLE-2 is the state-of-the-art speculative decoding method, improving on EAGLE with a dynamic draft tree. We introduce Dynamic Depth Decoding (DDD), which optimises EAGLE-2's tree drafting method using a dynamic depth. This extends the average speedup that EAGLE-2 achieves over EAGLE by $44\%$, giving DDD an average speedup of $3.16$x.