Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels
This addresses the problem of scaling RL data to web-scale levels for language model developers, enabling more efficient and capable models, though it is incremental as it builds on existing RL and data pipeline methods.
The paper tackles the data bottleneck in reinforcement learning (RL) for language models by introducing the Webscale-RL pipeline, which converts large-scale pre-training documents into 1.2 million diverse question-answer pairs, resulting in RL training that achieves performance comparable to continual pre-training with up to 100x fewer tokens.
Large Language Models (LLMs) have achieved remarkable success through imitation learning on vast text corpora, but this paradigm creates a training-generation gap and limits robust reasoning. Reinforcement learning (RL) offers a more data-efficient solution capable of bridging this gap, yet its application has been constrained by a critical data bottleneck: existing RL datasets are orders of magnitude smaller and less diverse than web-scale pre-training corpora. To address this, we introduce the Webscale-RL pipeline, a scalable data engine that systematically converts large-scale pre-training documents into millions of diverse, verifiable question-answer pairs for RL. Using this pipeline, we construct the Webscale-RL dataset, containing 1.2 million examples across more than 9 domains. Our experiments show that the model trained on this dataset significantly outperforms continual pretraining and strong data refinement baselines across a suite of benchmarks. Notably, RL training with our dataset proves substantially more efficient, achieving the performance of continual pre-training with up to 100$\times$ fewer tokens. Our work presents a viable path toward scaling RL to pre-training levels, enabling more capable and efficient language models.