Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents
For researchers training LLM agents on long-horizon tasks, SCPO provides a value-free reward-shaping method that improves learning from failed rollouts.
SCPO addresses semantic credit inconsistency in group-based RL for LLM agents by recovering step-level credit from successful siblings in the same rollout group, achieving 93.7% success on ALFWorld and 74.8% on WebShop at 1.5B parameters, with gains on hardest tasks.
Group-based reinforcement learning effectively post-trains LLM agents for long-horizon, sparse-reward tasks by deriving step-level credit from trajectory outcomes. However, this ties a step's credit to its rollout's final outcome: semantically near-identical intermediate steps receive opposite credit depending on whether their trajectory eventually succeeded or failed. Such semantic credit inconsistency sends conflicting gradients to similar actions and wastes the partially-correct progress inside failed rollouts. Motivated by this, we propose Semantic Consistency Policy Optimization (SCPO), a value-free reward-shaping method that mitigates this inconsistency by recovering step-level credit from successful siblings in the same rollout group. Concretely, SCPO scores each failed step against a successful sibling and adds positive step-level credit for new progress along that sibling. On ALFWorld and WebShop, SCPO matches or exceeds strong group-based baselines, reaching 93.7+/-4.1 percent success on ALFWorld and 74.8+/-2.0 percent on WebShop at 1.5B parameters, with gains concentrated on the hardest multi-step tasks.