R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning
It addresses a bottleneck in LLM reasoning for complex logic- and knowledge-intensive tasks, offering a novel method for optimizing reasoning-search trajectories.
The paper tackles the challenge of enabling large language models to effectively integrate reasoning with search interactions, proposing R-Search, a reinforcement learning framework that improves response quality in complex tasks, achieving performance gains of up to 32.2% in-domain and 25.1% out-of-domain over baselines.
Large language models (LLMs) have notably progressed in multi-step and long-chain reasoning. However, extending their reasoning capabilities to encompass deep interactions with search remains a non-trivial challenge, as models often fail to identify optimal reasoning-search interaction trajectories, resulting in suboptimal responses. We propose R-Search, a novel reinforcement learning framework for Reasoning-Search integration, designed to enable LLMs to autonomously execute multi-step reasoning with deep search interaction, and learn optimal reasoning search interaction trajectories via multi-reward signals, improving response quality in complex logic- and knowledge-intensive tasks. R-Search guides the LLM to dynamically decide when to retrieve or reason, while globally integrating key evidence to enhance deep knowledge interaction between reasoning and search. During RL training, R-Search provides multi-stage, multi-type rewards to jointly optimize the reasoning-search trajectory. Experiments on seven datasets show that R-Search outperforms advanced RAG baselines by up to 32.2% (in-domain) and 25.1% (out-of-domain). The code and data are available at https://github.com/QingFei1/R-Search.