QuRL: Efficient Reinforcement Learning with Quantized Rollout
This work addresses a specific efficiency problem in RL training for LLMs, offering incremental improvements for researchers and practitioners in AI.
The paper tackles the efficiency bottleneck of reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs), where the rollout process consumes up to 70% of training time, and proposes QuRL, a method using quantized actors to accelerate rollout, achieving 20% to 80% faster training.
Reinforcement learning with verifiable rewards (RLVR) has become a trending paradigm for training reasoning large language models (LLMs). However, due to the autoregressive decoding nature of LLMs, the rollout process becomes the efficiency bottleneck of RL training, consisting of up to 70\% of the total training time. In this work, we propose Quantized Reinforcement Learning (QuRL) that uses a quantized actor for accelerating the rollout. We address two challenges in QuRL. First, we propose Adaptive Clipping Range (ACR) that dynamically adjusts the clipping ratio based on the policy ratio between the full-precision actor and the quantized actor, which is essential for mitigating long-term training collapse. Second, we identify the weight update problem, where weight changes between RL steps are extremely small, making it difficult for the quantization operation to capture them effectively. We mitigate this problem through the invariant scaling technique that reduces quantization noise and increases weight update. We evaluate our method with INT8 and FP8 quantization experiments on DeepScaleR and DAPO, and achieve 20% to 80% faster rollout during training.