Training Domain Draft Models for Speculative Decoding: Best Practices and Insights
This work addresses the efficiency of speculative decoding for domain-specific applications, providing incremental guidelines for practitioners.
The paper tackles the problem of domain shift reducing acceptance rates in speculative decoding for large language models by systematically investigating knowledge distillation techniques to train domain-specific draft models, achieving performance improvements of 2% to 25% across domains and showing synthetic data can reach 80% to 93% of the effectiveness of historical user queries.
Speculative decoding is an effective method for accelerating inference of large language models (LLMs) by employing a small draft model to predict the output of a target model. However, when adapting speculative decoding to domain-specific target models, the acceptance rate of the generic draft model drops significantly due to domain shift. In this work, we systematically investigate knowledge distillation techniques for training domain draft models to improve their speculation accuracy. We compare white-box and black-box distillation approaches and explore their effectiveness in various data accessibility scenarios, including historical user queries, curated domain data, and synthetically generated alignment data. Our experiments across Function Calling, Biology, and Chinese domains show that offline distillation consistently outperforms online distillation by 11% to 25%, white-box distillation surpasses black-box distillation by 2% to 10%, and data scaling trends hold across domains. Additionally, we find that synthetic data can effectively align draft models and achieve 80% to 93% of the performance of training on historical user queries. These findings provide practical guidelines for training domain-specific draft models to improve speculative decoding efficiency.