SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding
This addresses the bottleneck of slow inference in LLMs for real-world deployment, though it is incremental as it builds on existing speculative decoding techniques.
The paper tackles the high inference latency of large language models by introducing SpecForge, an open-source framework for training speculative decoding models, which enables up to 9.9x faster training for Qwen3-235B-A22B and achieves up to 4.48x inference speedup with released draft models.
Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment.