Confidence-Modulated Speculative Decoding for Large Language Models
This work addresses the challenge of accelerating autoregressive inference for large language models, offering a plug-in method for more efficient and robust decoding under varying uncertainty conditions, though it is incremental as it builds on existing speculative decoding approaches.
The paper tackles the problem of inefficient speculative decoding in large language models by introducing a confidence-modulated framework that dynamically adjusts drafting lengths and verification criteria based on uncertainty measures, resulting in significant speedups while preserving or improving BLEU and ROUGE scores in machine translation and summarization tasks.
Speculative decoding has emerged as an effective approach for accelerating autoregressive inference by parallelizing token generation through a draft-then-verify paradigm. However, existing methods rely on static drafting lengths and rigid verification criteria, limiting their adaptability across varying model uncertainties and input complexities. This paper proposes an information-theoretic framework for speculative decoding based on confidence-modulated drafting. By leveraging entropy and margin-based uncertainty measures over the drafter's output distribution, the proposed method dynamically adjusts the number of speculatively generated tokens at each iteration. This adaptive mechanism reduces rollback frequency, improves resource utilization, and maintains output fidelity. Additionally, the verification process is modulated using the same confidence signals, enabling more flexible acceptance of drafted tokens without sacrificing generation quality. Experiments on machine translation and summarization tasks demonstrate significant speedups over standard speculative decoding while preserving or improving BLEU and ROUGE scores. The proposed approach offers a principled, plug-in method for efficient and robust decoding in large language models under varying conditions of uncertainty.