Less Noise, More Voice: Reinforcement Learning for Reasoning via Instruction Purification
This work addresses a specific bottleneck in RLVR for complex reasoning tasks, offering an incremental improvement in sampling efficiency and training stability.
The paper tackled the problem of inefficient exploration in reinforcement learning for LLM reasoning by identifying and removing interference tokens in prompts, resulting in a 3.88% average performance gain and over 1.6x speedup compared to baseline methods.
Reinforcement Learning with Verifiable Rewards (RLVR) has advanced LLM reasoning, but remains constrained by inefficient exploration under limited rollout budgets, leading to low sampling success and unstable training in complex tasks. We find that many exploration failures arise not from problem difficulty, but from a small number of prompt tokens that introduce interference. Building on this insight, we propose the Less Noise Sampling Framework (LENS), which first prompts by identifying and removing interference tokens. then transfers successful rollouts from the purification process to supervise policy optimization on the original noisy prompts, enabling the model to learn to ignore interference in the real-world, noisy prompting settings. Experimental results show that LENS significantly outperforms GRPO, delivering higher performance and faster convergence, with a 3.88% average gain and over 1.6$\times$ speedup. Our work highlights the critical role of pruning interference tokens in improving rollout efficiency, offering a new perspective for RLVR research.