Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization
This work addresses the credit assignment problem in reinforcement learning for search agents, offering a stable and effective training method that improves search behavior.
The paper proposes CW-GRPO, a framework that integrates process supervision into group relative policy optimization for training LLM-based search agents, achieving 5.0% and 6.3% improvements over standard GRPO on Qwen3-8B and Qwen3-1.7B respectively on knowledge-intensive benchmarks.
Search agents extend Large Language Models (LLMs) beyond static parametric knowledge by enabling access to up-to-date and long-tail information unavailable during pretraining. While reinforcement learning has been widely adopted for training such agents, existing approaches face key limitations: process supervision often suffers from unstable value estimation, whereas outcome supervision struggles with credit assignment due to sparse, trajectory-level rewards. To bridge this gap, we propose Contribution-Weighted GRPO (CW-GRPO), a framework that integrates process supervision into group relative policy optimization. Instead of directly optimizing process rewards, CW-GRPO employs an LLM judge to assess the retrieval utility and reasoning correctness at each search round, producing per-round contribution scores. These scores are used to rescale outcome-based advantages along the trajectory, enabling fine-grained credit assignment without sacrificing optimization stability. Experiments on multiple knowledge-intensive benchmarks show that CW-GRPO outperforms standard GRPO by 5.0% on Qwen3-8B and 6.3% on Qwen3-1.7B, leading to more effective search behaviors. Additional analysis reveals that successful trajectories exhibit concentrated contributions in specific rounds, providing empirical insight into search agent tasks.