LEDOM: An Open and Fundamental Reverse Language Model
This work presents a new paradigm in language modeling with potential broad applications, though it is incremental in exploring reverse processing as a foundational approach.
The authors tackled the problem of language modeling by introducing LEDOM, the first purely reverse language model trained on 435B tokens, which processes sequences in reverse order and achieves substantial performance improvements on mathematical reasoning tasks through a novel Reverse Reward application.
We introduce LEDOM, the first purely reverse language model, trained autoregressively on 435B tokens with 2B and 7B parameter variants, which processes sequences in reverse temporal order through previous token prediction. For the first time, we present the reverse language model as a potential foundational model across general tasks, accompanied by a set of intriguing examples and insights. Based on LEDOM, we further introduce a novel application: Reverse Reward, where LEDOM-guided reranking of forward language model outputs leads to substantial performance improvements on mathematical reasoning tasks. This approach leverages LEDOM's unique backward reasoning capability to refine generation quality through posterior evaluation. Our findings suggest that LEDOM exhibits unique characteristics with broad application potential. We will release all models, training code, and pre-training data to facilitate future research.