OffSeeker: Online Reinforcement Learning Is Not All You Need for Deep Research Agents
This work addresses the financial inefficiency of training deep research agents, offering a more accessible offline alternative for researchers and developers.
The paper tackles the high cost of online reinforcement learning for deep research agents by introducing an open-source suite for offline training, including a task synthesis framework and curated datasets, and shows that their 8B-parameter model, OffSeeker, achieves competitive performance with larger models trained online across six benchmarks.
Deep research agents have shown remarkable potential in handling long-horizon tasks. However, state-of-the-art performance typically relies on online reinforcement learning (RL), which is financially expensive due to extensive API calls. While offline training offers a more efficient alternative, its progress is hindered by the scarcity of high-quality research trajectories. In this paper, we demonstrate that expensive online reinforcement learning is not all you need to build powerful research agents. To bridge this gap, we introduce a fully open-source suite designed for effective offline training. Our core contributions include DeepForge, a ready-to-use task synthesis framework that generates large-scale research queries without heavy preprocessing; and a curated collection of 66k QA pairs, 33k SFT trajectories, and 21k DPO pairs. Leveraging these resources, we train OffSeeker (8B), a model developed entirely offline. Extensive evaluations across six benchmarks show that OffSeeker not only leads among similar-sized agents but also remains competitive with 30B-parameter systems trained via heavy online RL.