Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decoding
This work addresses a critical bottleneck in accelerating LLM inference for real-world deployment, though it is incremental as it builds on existing heuristic-based methods.
The paper tackles the problem of optimizing speculation length in speculative decoding for LLM inference, introducing GammaTune and GammaTune+ algorithms that achieve average speedups of 15% and 16% respectively, while reducing performance variance.
Speculative decoding accelerates large language model (LLM) inference by using a smaller draft model to propose tokens, which are then verified by a larger target model. However, selecting an optimal speculation length is critical for maximizing speedup while minimizing wasted computation. We introduce \textit{GammaTune} and \textit{GammaTune+}, training-free adaptive algorithms that dynamically adjust speculation length based on token acceptance rates using a heuristic-based switching mechanism. Evaluated on SpecBench across multiple tasks and model pairs, our method outperforms other heuristic-based approaches and fixed-length speculative decoding, achieving an average speedup of 15\% ($\pm$5\%) with \textit{GammaTune} and 16\% ($\pm$3\%) with \textit{GammaTune+}, while reducing performance variance. This makes \textit{GammaTune} a robust and efficient solution for real-world deployment.