VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models
This work addresses the challenge of optimizing LLM training efficiency for mathematical reasoning, offering an incremental improvement over existing reinforcement learning approaches by incorporating curriculum learning principles.
The paper tackles the problem of improving large language models (LLMs) on mathematical reasoning tasks by addressing the lack of explicit consideration for sample difficulty in existing reinforcement learning methods, proposing VCRL, a curriculum reinforcement learning framework that dynamically adjusts training sample difficulty based on reward variance, which outperforms current baselines on five benchmarks and two models.
Policy-based reinforcement learning currently plays an important role in improving LLMs on mathematical reasoning tasks. However, existing rollout-based reinforcement learning methods (GRPO, DAPO, GSPO, etc.) fail to explicitly consider LLMs' learning ability for samples of different difficulty levels, which is contrary to the human cognitive process of mathematical reasoning tasks from easy to difficult. Intuitively, we find that the variance of the rollout group's reward in RLVR partly reflects the difficulty of the current sample for LLMs. Samples that are too easy or too difficult have a lower variance, while samples with moderate difficulty have a higher variance. Based on this, we propose VCRL, a curriculum reinforcement learning framework that dynamically controls the difficulty of training samples based on the variance of group rewards. Experiments on five mathematical benchmarks and two models reveal the advantages of VCRL over the current LLM RL baselines.