Experience-Driven Dynamic Exits for LLMs with Reinforcement Learning
For practitioners deploying large language models, this work offers a practical method to accelerate inference without sacrificing quality, though the gains are incremental over existing speculative decoding approaches.
LEDE uses offline reinforcement learning to dynamically select exit layers and speculation lengths during LLM inference, achieving 2.0-2.7x speedup over autoregressive decoding and 17% additional speedup over static speculative decoding.
Large Language Models suffer from slow autoregressive inference. While self-speculative decoding accelerates this process, its efficiency is hampered by static configurations like fixed exit layers and speculation lengths. We reframe this optimization as a \textbf{Markov Decision Process} and propose \textbf{LEDE}, a framework that uses offline reinforcement learning. LEDE learns a policy to dynamically select the optimal exit layer and speculation length based on the local context of the generated sequence at each step, balancing computational cost and draft quality. Comprehensive evaluations on Llama-2 and Llama-3 models show LEDE achieves up to a $2.0\times$$\sim$$2.7\times$ speedup over autoregressive decoding and and provides an additional 17\% speedup over the static speculative baselines.