MineDraft: A Framework for Batch Parallel Speculative Decoding
This addresses a bottleneck in production-ready inference systems for AI applications, offering a practical improvement over existing speculative decoding techniques.
The paper tackles the inefficiency of speculative decoding in large language model inference by proposing MineDraft, a batch parallel framework that overlaps drafting and verification stages, resulting in up to 75% higher throughput and 39% lower latency compared to standard methods.
Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to propose draft tokens that are subsequently verified by a larger target model. However, the performance of standard SD is often limited by the strictly sequential execution of these drafting and verification stages. To address this, this paper proposes MineDraft, a batch parallel speculative decoding (PSD) framework designed to effectively hide drafting latency by overlapping it with verification. Our theoretical analysis shows that PSD is substantially more efficient than standard SD. MineDraft realizes the PSD through a novel batch-parallel design that maintains two batches of requests, overlapping drafting for one batch with verification for the other. Our experimental results show significant improvements of MineDraft in both throughput (up to 75%) and end-to-end latency (up to 39%) over standard SD. Furthermore, we have implemented MineDraft as a plugin for vLLM, demonstrating its practicality for production-ready inference systems.