Fast LLM Post-training via Decoupled and Best-of-N Speculation
This addresses a bottleneck in LLM post-training efficiency, offering incremental improvements for researchers and practitioners optimizing training workflows.
The paper tackles the problem of slow rollout in large language model post-training by introducing SpecActor, which uses dynamic decoupled speculation and dynamic Best-of-N speculation to accelerate token generation, achieving speedups of 1.3–1.7x over baselines and 1.3–1.5x over naive speculative decoding.
Rollout dominates the training time in large language model (LLM) post-training, where the trained model is used to generate tokens given a batch of prompts. SpecActor achieves fast rollout with speculative decoding that deploys a fast path (e.g., a smaller model) to accelerate the unparallelizable generation, while the correctness is guaranteed by fast parallel verification of the outputs with the original model. SpecActor addresses two foundational challenges in speculative rollout by (1) a \emph{dynamic decoupled speculation} execution method that maximizes the GPU computational efficiency to realize speedup for large-batch execution -- a configuration common in training but unfriendly to speculative execution and (2) a \emph{dynamic Best-of-N speculation} method that selects and combines different drafting methods according to the rollout progress. It substantially improves the speculation accuracy even when the best drafting method is unknown a priori, meanwhile without requiring adding extra computation resources. {\sys} is {1.3--1.7}\,$\times$ faster than common post-training baselines, and is {1.3--1.5}\,$\times$ faster compared to naively adopting speculative decoding for rollout.