VeriGate: Verifier-Gated Step-Level Supervision for GRPO
This work provides a substantial improvement in training reasoning models for the machine learning community, particularly those using verifier-based policy optimization, by making learning more robust and efficient.
This paper introduces VeriGate, a method that enhances Group Relative Policy Optimization (GRPO) by integrating step-level supervision. VeriGate addresses the limitations of sparse outcome-only rewards in GRPO, such as zero-gradient failures and limited exploration, by selectively applying process supervision and using future-cumulated, group-normalized token-level advantages. Empirically, VeriGate improves average accuracy by approximately 20% for 1.5B models and 12% for 7B models on six reasoning benchmarks.
Group Relative Policy Optimization (GRPO) is an effective recipe for training reasoning models with verifier-based outcome rewards, but its supervision is sparse: when all sampled trajectories for a prompt receive the same verifier reward, the group-relative advantage collapses to zero and learning stalls. Outcome-only rewards also provide no step-level credit assignment, limiting exploration and making it harder to learn robust reasoning. We present VeriGate (Verifier-Gated Step-Level GRPO), a verifier-gated extension of GRPO that addresses these limitations with three design choices. First, VeriGate keeps the verifier in charge whenever verifier rewards induce a meaningful preference among sampled trajectories, and uses process supervision only when verifier rewards are degenerate. Second, instead of collapsing Process Reward Model (PRM) step scores into a single trajectory reward, VeriGate converts them into future-cumulated rewards to assign continuation-aware credit. Third, VeriGate transforms these rewards into group-normalized token-level advantages, restoring informative gradients and fine-grained credit assignment while remaining less susceptible to reward hacking than methods that optimize aggregated PRM scores. Empirically, training on MATH with 1.5B and 7B Qwen2.5-Instruct models and evaluating on six reasoning benchmarks, VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively, substantially reduces zero-gradient failures, decreases reward-hacking behavior, and improves reasoning quality relative to outcome-only GRPO and PRM-as-outcome baselines.