It's Not You, It's Clipping: A Soft Trust-Region via Probability Smoothing for LLM RL
This work addresses a specific bottleneck in RL fine-tuning of LLMs, offering an incremental improvement over existing methods like GRPO.
The paper tackles the problem of training large language models with reinforcement learning by addressing the limitations of ratio clipping, which discards information and causes gradient discontinuities. The proposed Probability Smoothing Policy Optimisation (PSPO) method improves performance substantially, achieving over 20% boosts on GSM8K (e.g., 39.7% vs. 17.6% for a 0.5B model).
Training large language models (LLMs) with reinforcement learning (RL) methods such as PPO and GRPO commonly relies on ratio clipping to stabilise updates. While effective at preventing instability, clipping discards information and introduces gradient discontinuities. We propose Probability Smoothing Policy Optimisation (PSPO), which smooths the current policy's probabilities toward the old (behaviour) policy before computing the importance ratio, analogous to label smoothing. Unlike clipping, PSPO preserves gradient signal, while interpolation toward the old policy creates a soft trust region that discourages large, destabilising updates, with formal guarantees. We instantiate PSPO within GRPO (GR-PSPO) and fine-tune Qwen2.5-0.5B and Qwen2.5-1.5B on GSM8K, evaluating on GSM8K test and the cross-dataset generalisation on SVAMP, ASDiv, and MATH-500. Relative to unclipped GRPO (single iteration; no data reuse, ratio always = 1), GR-PSPO achieves similar performance but improves the reasoning leading to clearer and more concise responses which are more logical. Compared to clipped GRPO, GR-PSPO substantially improves performance both the 0.5B and 1.5B models, with a boost of over 20% on GSM8K (39.7% vs. 17.6% for 0.5B, 59.4% vs. 37.8% for 1.5B).