WebSeer: Training Deeper Search Agents through Reinforcement Learning with Self-Reflection
This addresses the need for more intelligent search agents in web-based environments, though it appears incremental as it builds on existing reinforcement learning methods with a novel self-reflection enhancement.
The paper tackles the problem of shallow tool-use depth and error accumulation in search agents by introducing WebSeer, a reinforcement learning agent with self-reflection, which achieves state-of-the-art accuracies of 72.3% on HotpotQA and 90.0% on SimpleQA.
Search agents have achieved significant advancements in enabling intelligent information retrieval and decision-making within interactive environments. Although reinforcement learning has been employed to train agentic models capable of more dynamic interactive retrieval, existing methods are limited by shallow tool-use depth and the accumulation of errors over multiple iterative interactions. In this paper, we present WebSeer, a more intelligent search agent trained via reinforcement learning enhanced with a self-reflection mechanism. Specifically, we construct a large dataset annotated with reflection patterns and design a two-stage training framework that unifies cold start and reinforcement learning within the self-reflection paradigm for real-world web-based environments, which enables the model to generate longer and more reflective tool-use trajectories. Our approach substantially extends tool-use chains and improves answer accuracy. Using a single 14B model, we achieve state-of-the-art results on HotpotQA and SimpleQA, with accuracies of 72.3% and 90.0%, respectively, and demonstrate strong generalization to out-of-distribution datasets. The code is available at https://github.com/99hgz/WebSeer