Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning
This addresses the challenge of selecting appropriate difficulty problems for efficient RORL training, which is an incremental improvement over existing curriculum learning and filtering methods.
The paper tackles the problem of reward sparsity in Reasoning-Oriented Reinforcement Learning (RORL) by proposing balanced online difficulty filtering, which curates training batches with problems of intermediate difficulty to maximize training effectiveness. Experimental results show this method yields an additional 10% improvement in AIME and 4% average improvement over plain GRPO across five math reasoning benchmarks.
Reasoning-Oriented Reinforcement Learning (RORL) enhances the reasoning ability of Large Language Models (LLMs). However, due to the sparsity of rewards in RORL, effective training is highly dependent on the selection of problems of appropriate difficulty. Although curriculum learning attempts to address this by adjusting difficulty, it often relies on static schedules, and even recent online filtering methods lack theoretical grounding and a systematic understanding of their effectiveness. In this work, we theoretically and empirically show that curating the batch with the problems that the training model achieves intermediate accuracy on the fly can maximize the effectiveness of RORL training, namely balanced online difficulty filtering. We first derive that the lower bound of the KL divergence between the initial and the optimal policy can be expressed with the variance of the sampled accuracy. Building on those insights, we show that balanced filtering can maximize the lower bound, leading to better performance. Experimental results across five challenging math reasoning benchmarks show that balanced online filtering yields an additional 10% in AIME and 4% improvements in average over plain GRPO. Moreover, further analysis shows the gains in sample efficiency and training time efficiency, exceeding the maximum reward of plain GRPO within 60% training time and the volume of the training set.