RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
This paper provides evidence that RL can be applied earlier in LLM training, potentially simplifying the pipeline and improving performance, which is significant for practitioners developing LLM training strategies.
The authors challenge the standard LLM training pipeline by applying RL directly to intermediate pre-training checkpoints, finding that RL is effective early and often matches the full SFT→RL pipeline. They show that targeted pre-training data composition is a strong lever for RL effectiveness, and that merging RL and SFT objectives via parallel averaging outperforms all other methods across metrics while preserving general capabilities.
The standard LLM training pipeline applies reinforcement learning (RL) only after pre-training and supervised fine-tuning (SFT). We question this status quo by training a LLM from scratch and applying RL, SFT, and SFT followed by RL directly to intermediate pre-training checkpoints. We find that RL is effective very early, and often matches the full SFT$\to$RL pipeline early as well. Through experiments on harder problems, we find that targeted pre-training data composition is a strong lever for RL effectiveness, even more so than model scale. Beyond reasoning accuracy, applying RL directly to base checkpoints expands the model's distribution; the sharpening effect reported in recent work arises only when RL follows SFT. The general capabilities of the model remain essentially unchanged by RL, while they degrade following SFT. Finally, we merge RL and SFT objectives by parallel averaging, which outperforms across all other training methods discussed, across metrics, while preserving general capabilities. Together, these results suggest that LLM training might benefit from an expanded use of RL.