Clip Your Sequences Fairly: Enforcing Length Fairness for Sequence-Level RL
This addresses a fairness issue in RL for LLMs, improving training stability and performance, though it is incremental as it modifies an existing clipping approach.
The paper tackles the problem of length bias in sequence-level reinforcement learning for LLMs, where fixed clipping ranges distort optimization by unfairly reweighting short versus long responses; the proposed FSPO method enforces length-fair clipping, stabilizing training and achieving performance gains, such as on the Qwen3-8B-Base model.
We propose FSPO (Fair Sequence Policy Optimization), a sequence-level reinforcement learning method for LLMs that enforces length-fair clipping on the importance-sampling (IS) weight. We study RL methods with sequence-level IS and identify a mismatch when PPO/GRPO-style clipping is transplanted to sequences: a fixed clip range systematically reweights short vs. long responses, distorting the optimization direction. FSPO introduces a simple remedy: we clip the sequence log-IS ratio with a band that scales as $\sqrt{L}$. Theoretically, we formalize length fairness via a Length Reweighting Error (LRE) and prove that small LRE yields a cosine directional guarantee between the clipped and true updates. Empirically, FSPO flattens clip rates across length bins, stabilizes training, and outperforms baselines across model sizes and evaluation datasets, with the largest gains on the Qwen3-8B-Base model.