SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models
For researchers training reasoning-oriented LLMs with RL, SPS addresses the exploration bottleneck that limits multi-sample performance, offering a method to enhance diversity without external supervision.
The paper identifies that RL training for LLMs concentrates probability mass on a narrow set of high-reward trajectories, limiting exploration and multi-sample performance (Pass@k). They propose SPS, which interleaves RL with inverse reinforcement learning to reshape trajectory distributions, improving Pass@k on five reasoning benchmarks.
Reinforcement learning (RL) has emerged as a promising paradigm for training reasoning-oriented models by leveraging rule-based reward signals. However, RL training typically tends to improve single-sample success rates (i.e., Pass@1) while offering limited exploration of diverse reasoning trajectories, which is crucial for multi-sample performance (i.e., Pass@k). Our preliminary analysis reveals that this limitation stems from a fundamental squeezing effect, whereby probability mass is excessively concentrated on a narrow subset of high-reward trajectories, restricting genuine exploration and constraining attainable performance under RL training. To address this issue, in this work, we propose Steering Probability Squeezing (SPS), a training paradigm that interleaves conventional RL with inverse reinforcement learning (IRL). SPS treats on-policy rollouts as demonstrations and employs IRL to explicitly reshape the induced trajectory distribution, thereby enhancing exploration without introducing external supervision. Experiments on five commonly used reasoning benchmarks demonstrate that SPS can enable better exploration and improve Pass@k. Beyond algorithmic contributions, we provide an analysis of RL learning dynamics and identify an empirical upper bound on Pass@k, shedding light on intrinsic exploration limits in RL-based reasoning models. Our findings suggest that alternating between RL and IRL offers an effective pathway toward extending the exploration capacity of reasoning-oriented large language models.