PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length
This work addresses a bottleneck in accelerating inference for large language models, offering incremental improvements in decoding speed for AI applications.
The paper tackles the problem of inefficient fixed draft lengths in speculative decoding for large language models by proposing Pacer, which dynamically controls draft length using a blockwise pre-verification layer, achieving up to 2.66x speedup over autoregressive decoding and outperforming standard speculative decoding.
Speculative decoding (SD) is a powerful technique for accelerating the inference process of large language models (LLMs) without sacrificing accuracy. Typically, SD employs a small draft model to generate a fixed number of draft tokens, which are then verified in parallel by the target model. However, our experiments reveal that the optimal draft length varies significantly across different decoding steps. This variation suggests that using a fixed draft length limits the potential for further improvements in decoding speed. To address this challenge, we propose Pacer, a novel approach that dynamically controls draft length using a lightweight, trainable pre-verification layer. This layer pre-verifies draft tokens blockwise before they are sent to the target model, allowing the draft model to stop token generation if the blockwise pre-verification fails. We implement Pacer on multiple SD model pairs and evaluate its performance across various benchmarks. Our results demonstrate that Pacer achieves up to 2.66x Speedup over autoregressive decoding and consistently outperforms standard speculative decoding. Furthermore, when integrated with Ouroboros, Pacer attains up to 3.09x Speedup.