PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation
This addresses efficiency bottlenecks in RL for LLMs, offering a scalable solution for researchers and practitioners, though it is incremental in improving existing methods.
The paper tackles the challenge of scaling reinforcement learning for large language models by introducing PipelineRL, which achieves about 2x faster learning on long-form reasoning tasks using 128 H100 GPUs while maintaining highly on-policy training data.
Reinforcement Learning (RL) is increasingly utilized to enhance the reasoning capabilities of Large Language Models (LLMs). However, effectively scaling these RL methods presents significant challenges, primarily due to the difficulty in maintaining high AI accelerator utilization without generating stale, off-policy data that harms common RL algorithms. This paper introduces PipelineRL, an approach designed to achieve a superior trade-off between hardware efficiency and data on-policyness for LLM training. PipelineRL employs concurrent asynchronous data generation and model training, distinguished by the novel in-flight weight updates. This mechanism allows the LLM generation engine to receive updated model weights with minimal interruption during the generation of token sequences, thereby maximizing both the accelerator utilization and the freshness of training data. Experiments conducted on long-form reasoning tasks using 128 H100 GPUs demonstrate that PipelineRL achieves approximately $\sim 2x$ faster learning compared to conventional RL baselines while maintaining highly on-policy training data. A scalable and modular open-source implementation of PipelineRL is also released as a key contribution.