TapOut: A Bandit-Based Approach to Dynamic Speculative Decoding
This addresses the problem of inefficient speculative decoding for LLM users by providing a plug-and-play solution, though it is incremental as it builds on existing dynamic speculation methods.
The paper tackled the challenge of determining the optimal number of tokens to draft in dynamic speculative decoding for LLMs, proposing TapOut, a bandit-based approach that achieved competitive or superior speedups without hyperparameter tuning across diverse models and datasets.
Speculative decoding accelerates LLMs by using a lightweight draft model to generate tokens autoregressively before verifying them in parallel with a larger target model. However, determining the optimal number of tokens to draft remains a key challenge limiting the approach's effectiveness. Dynamic speculative decoding aims to intelligently decide how many tokens to draft to achieve maximum speedups. Existing methods often rely on hand-tuned, sensitive thresholds (e.g., token entropy), which are costly to set and generalize poorly across models and domains. We propose TapOut, an online, training-free, plug-and-play algorithm for dynamic speculation policy selection using multi-armed bandits. Our approach employs a meta-algorithm that selects among multiple parameter-free dynamic speculation strategies based on past reward and exploration. We conduct extensive experiments across diverse model pairs and datasets, showing that TapOut achieves competitive or superior speedups compared to well-established dynamic speculation baselines without any hyperparameter tuning.