Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony
This work addresses scalability issues in RL post-training for LLMs, offering incremental improvements in system efficiency for AI researchers and practitioners.
The paper tackles the problem of low resource utilization and limited scalability in synchronous Reinforcement Learning (RL) post-training for Large Language Models (LLMs) by introducing ROLL Flash, a system with asynchronous RL post-training, achieving up to 2.24x speedup on RLVR tasks and 2.72x on agentic tasks with the same GPU budget.
Synchronous Reinforcement Learning (RL) post-training has emerged as a crucial step for enhancing Large Language Models (LLMs) with diverse capabilities. However, many systems designed to accelerate RL post-training still suffer from low resource utilization and limited scalability. We present ROLL Flash, a system that extends ROLL with native support for asynchronous RL post-training. ROLL Flash is built upon two core design principles: fine-grained parallelism and rollout-train decoupling. Guided by these principles, ROLL Flash provides flexible programming interfaces that enable a fully asynchronous training architecture and support efficient rollout mechanisms, including queue scheduling and environment-level asynchronous execution. Through comprehensive theoretical analysis and extensive experiments, we demonstrate that ROLL Flash significantly improves resource utilization and scalability over synchronous RL post-training. ROLL Flash achieves up to 2.24x speedup on RLVR tasks and 2.72x on agentic tasks, using the same GPU budget as synchronous baselines. Furthermore, we implement several popular off-policy algorithms and verify that asynchronous training can achieve performance on par with synchronous training.