DeepDive: Advancing Deep Search Agents with Knowledge Graphs and Multi-Turn RL
This work addresses the challenge of enhancing deep search agents for complex real-world tasks, representing an incremental improvement in a domain-specific area.
The paper tackles the problem of poor performance of open LLMs as deep search agents due to limited long-horizon reasoning and lack of difficult data, by synthesizing complex questions from knowledge graphs and using multi-turn RL, resulting in DeepDive-32B achieving a new open-source competitive result on BrowseComp and outperforming other models.
Augmenting large language models (LLMs) with browsing tools substantially improves their potential as deep search agents to solve complex, real-world tasks. Yet, open LLMs still perform poorly in such settings due to limited long-horizon reasoning capacity with browsing tools and the lack of sufficiently difficult supervised data. To address these challenges, we present DeepDive to advance deep search agents. First, we propose a strategy to automatically synthesize complex, difficult, and hard-to-find questions from open knowledge graphs. Second, we apply end-to-end multi-turn reinforcement learning (RL) to enhance LLMs' long-horizon reasoning with deep search. To encourage diversity and reduce redundancy, we design a redundancy penalty that discourages repeated similar queries. Experiments show that DeepDive-32B achieves a new open-source competitive result on BrowseComp, outperforming WebSailor, DeepSeek-R1-Browse, and Search-o1. We demonstrate that multi-turn RL training improves deep search ability and significantly contributes to the performance improvements across multiple benchmarks. We observe that DeepDive enables test-time scaling of tool calls and parallel sampling. All datasets, models, and code are publicly available at https://github.com/THUDM/DeepDive.