PROGRESS: Coverage-guided RL to Train Search-augmented LLM Agent
For researchers working on search-augmented LLM agents, this work addresses the lack of supervision over search behavior, potentially improving complex query decomposition and task performance.
The paper proposes PROGRESS, a coverage-guided reinforcement learning method that uses teacher models to provide coverage rewards for decomposed query generation in search-augmented LLM agents. Experiments show that this approach improves overall task performance compared to outcome-level reward baselines.
Existing search-augmented LLM agents are trained using Reinforcement Learning to boost its reasoning capabilities. However, these approaches primarily rely on outcome-level rewards, which provide little supervision over search behavior and overlook agent's ability to decompose complex queries properly. To mitigate this issue, we propose PROGRESS which utilizes teacher-guided coverage reward to explicitly shape decomposed query generation of the policy model. During training, frozen teacher models are used to decompose complex queries into essential search queries. These essential search queries are utilized to guide the search behavior of the policy model. Integrated into an R1-style training framework, our approach provides lightweight guidance over query decomposition decisions without dense process-level supervision. Experiments show that coverage-guided RL improves overall task performance, highlighting the importance of explicitly supervising search strategies in agentic LLMs.