Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simple
This work addresses the costly experimental optimization of inference throughput for speculative decoding, offering a theoretical solution for researchers and practitioners in efficient LLM deployment.
The paper tackles the problem of optimizing throughput in speculative decoding inference systems by developing a theory that analytically links pre-trained LLM hyperparameters to throughput efficiency, enabling prediction of optimal hyperparameters before pre-training.
Speculative decoding is a technique that uses multiple language models to accelerate infer- ence. Previous works have used an experi- mental approach to optimize the throughput of the inference pipeline, which involves LLM training and can be costly. This study of spec- ulative decoding proposes a theory that ana- lytically connects the key hyperparameters of pre-trained LLMs to the throughput efficiency of a downstream SD-based inference system. The theory allows the prediction of throughput- optimal hyperparameters for the components of an inference system before their pre-training.