Towards Fast Multilingual LLM Inference: Speculative Decoding and Specialized Drafters
This work addresses inference efficiency for multilingual LLM deployment, representing an incremental improvement with domain-specific impact.
The paper tackles the problem of high inference time for multilingual large language models by proposing a training recipe for language-specific draft models in speculative decoding, achieving substantial speedup compared to previous methods as validated across various languages and evaluations.
Large language models (LLMs) have revolutionized natural language processing and broadened their applicability across diverse commercial applications. However, the deployment of these models is constrained by high inference time in multilingual settings. To mitigate this challenge, this paper explores a training recipe of an assistant model in speculative decoding, which is leveraged to draft and-then its future tokens are verified by the target LLM. We show that language-specific draft models, optimized through a targeted pretrain-and-finetune strategy, substantially brings a speedup in inference time compared to the previous methods. We validate these models across various languages in inference time, out-of-domain speedup, and GPT-4o evaluation.