Multi-Token Prediction via Self-Distillation
This provides a simpler and faster inference method for language model users, avoiding the complexity of auxiliary models, though it is incremental as it builds on existing distillation and multi-token prediction ideas.
The paper tackles the problem of slow autoregressive language model inference by converting a pretrained model into a standalone multi-token predictor using online distillation, achieving over 3x faster decoding on GSM8K with less than a 5% accuracy drop.
Existing techniques for accelerating language model inference, such as speculative decoding, require training auxiliary speculator models and building and deploying complex inference pipelines. We consider a new approach for converting a pretrained autoregressive language model from a slow single next token prediction model into a fast standalone multi-token prediction model using a simple online distillation objective. The final model retains the exact same implementation as the pretrained initial checkpoint and is deployable without the addition of any auxiliary verifier or other specialized inference code. On GSM8K, our method produces models that can decode more than $3\times$ faster on average at $<5\%$ drop in accuracy relative to single token decoding performance.