Predicting the Order of Upcoming Tokens Improves Language Modeling
This addresses a bottleneck in language model training for NLP researchers and practitioners, offering a more effective auxiliary objective.
The paper tackles the problem of inconsistent improvements in language modeling from multi-token prediction by proposing token order prediction, which trains models to order upcoming tokens instead of predicting them exactly, resulting in overall outperformance over standard and multi-token prediction on eight NLP benchmarks.
Multi-Token Prediction (MTP) has been proposed as an auxiliary objective to improve next-token prediction (NTP) in language model training but shows inconsistent improvements, underperforming in standard NLP benchmarks. We argue that MTP's exact future token prediction is too difficult as an auxiliary loss. Instead, we propose Token Order Prediction (TOP), which trains models to order upcoming tokens by their proximity using a learning-to-rank loss. TOP requires only a single additional unembedding layer compared to MTP's multiple transformer layers. We pretrain models of 340M, 1.8B, and 7B parameters using NTP, MTP, and TOP objectives. Results on eight standard NLP benchmarks show that TOP overall outperforms both NTP and MTP even at scale. Our code is available at https://github.com/zaydzuhri/token-order-prediction