Skip-Connected Policy Optimization for Implicit Advantage
For practitioners of reinforcement learning for reasoning, SKPO provides a practical method to leverage dense rewards without the variance penalty, improving performance on multiple benchmarks.
GRPO's Monte Carlo advantage estimation for early reasoning tokens is high-variance under limited sampling, making dense rewards worse than outcome-only rewards. SKPO decomposes reasoning into upstream and downstream phases with a skip connection, achieving 3.91% and 6.17% relative gains over baselines on Qwen2.5-Math-7B and Llama-3.2-3B across math, reasoning, and code tasks.
Group Relative Policy Optimization (GRPO) has proven effective in RLVR by using outcome-based rewards. While fine-grained dense rewards can theoretically improve performance, we reveal that under practical sampling budgets, Monte Carlo estimation yields high-variance and sign-inconsistent advantages for early reasoning tokens, paradoxically underperforming outcome-only GRPO. We propose Skip-Connected Optimization (SKPO), which decomposes reasoning into upstream and downstream phases: upstream receives dense rewards from downstream Monte Carlo sampling with single-stream optimization; downstream maintains group-relative optimization, where a skip connection concatenates the upstream segment with the original problem, enabling the model to leverage helpful upstream reasoning while preserving the freedom to bypass flawed reasoning through direct problem access. Experiments demonstrate improvements of 3.91% and 6.17% relative gains over the strongest baselines on Qwen2.5-Math-7B and Llama-3.2-3B respectively across mathematical benchmarks and out-of-domain tasks including general reasoning and code generation. Further analysis reveals an implicit advantage: SKPO generates trajectories with higher intermediate-step quality even when matched for final correctness.