On multi-token prediction for efficient LLM inference
This work addresses the problem of accelerating LLM inference for AI practitioners, but it is incremental as it builds on existing methods and highlights limitations without achieving a breakthrough.
The paper investigates multi-token prediction (MTP) in LLMs pre-trained for next-token prediction, finding that these models have inherent MTP capabilities that improve with scale, but integrating MTP heads is challenging due to specialization, and joint training helps but does not fully overcome this barrier.
We systematically investigate multi-token prediction (MTP) capabilities within LLMs pre-trained for next-token prediction (NTP). We first show that such models inherently possess MTP capabilities via numerical marginalization over intermediate token probabilities, though performance is data-dependent and improves with model scale. Furthermore, we explore the challenges of integrating MTP heads into frozen LLMs and find that their hidden layers are strongly specialized for NTP, making adaptation non-trivial. Finally, we show that while joint training of MTP heads with the backbone improves performance, it cannot fully overcome this barrier, prompting further research in this direction. Our findings provide a deeper understanding of MTP applied to pretrained LLMs, informing strategies for accelerating inference through parallel token prediction.