MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification
This work addresses a fundamental bottleneck in accelerating large language model inference for practical applications, though it is incremental as it modifies only the verification rule within existing frameworks.
The paper tackles the inefficiency in speculative decoding verification by proposing a margin-aware verification strategy that adapts to the target model's decisiveness, achieving consistent and significant inference speedups across model scales from 8B to 235B while preserving generation quality.
Speculative Decoding (SD) accelerates autoregressive large language model (LLM) inference by decoupling generation and verification. While recent methods improve draft quality by tightly coupling the drafter with the target model, the verification mechanism itself remains largely unchanged, relying on strict token-level rejection sampling. In practice, modern LLMs frequently operate in low-margin regimes where the target model exhibits weak preference among top candidates. In such cases, rejecting plausible runner-up tokens yields negligible information gain while incurring substantial rollback cost, leading to a fundamental inefficiency in verification. We propose Margin-Aware Speculative Verification, a training-free and domain-agnostic verification strategy that adapts to the target model's local decisiveness. Our method conditions verification on decision stability measured directly from the target logits and relaxes rejection only when strict verification provides minimal benefit. Importantly, the approach modifies only the verification rule and is fully compatible with existing target-coupled speculative decoding frameworks. Extensive experiments across model scales ranging from 8B to 235B demonstrate that our method delivers consistent and significant inference speedups over state-of-the-art baselines while preserving generation quality across diverse benchmarks.