ReMiT: RL-Guided Mid-Training for Iterative LLM Evolution
This work addresses the need for more efficient and effective iterative evolution of LLMs, offering a novel approach to enhance reasoning capabilities across domains like math and code, though it appears incremental in building on existing training phases.
The paper tackles the problem of unidirectional training pipelines for large language models by proposing a bidirectional process where reinforcement learning-tuned models retroactively improve the pre-trained foundation, resulting in an average 3% improvement on 10 pre-training benchmarks and sustained gains of over 2% throughout post-training.
Standard training pipelines for large language models (LLMs) are typically unidirectional, progressing from pre-training to post-training. However, the potential for a bidirectional process--where insights from post-training retroactively improve the pre-trained foundation--remains unexplored. We aim to establish a self-reinforcing flywheel: a cycle in which reinforcement learning (RL)-tuned model strengthens the base model, which in turn enhances subsequent post-training performance, requiring no specially trained teacher or reference model. To realize this, we analyze training dynamics and identify the mid-training (annealing) phase as a critical turning point for model capabilities. This phase typically occurs at the end of pre-training, utilizing high-quality corpora under a rapidly decaying learning rate. Building upon this insight, we introduce ReMiT (Reinforcement Learning-Guided Mid-Training). Specifically, ReMiT leverages the reasoning priors of RL-tuned models to dynamically reweight tokens during the mid-training phase, prioritizing those pivotal for reasoning. Empirically, ReMiT achieves an average improvement of 3\% on 10 pre-training benchmarks, spanning math, code, and general reasoning, and sustains these gains by over 2\% throughout the post-training pipeline. These results validate an iterative feedback loop, enabling continuous and self-reinforcing evolution of LLMs.