Training Documents Reranker with Search Rubrics for Deep Research Agent
This work aims to improve the quality of retrieved document sets for deep research agents by explicitly defining and enforcing complex information needs, which is an incremental improvement for researchers and agents relying on retrieval-augmented generation.
This paper addresses the limitation of existing document retrievers for deep research agents, which often fail to provide a set of documents that collectively satisfy complex information needs. The authors propose search-oriented rubrics, explicitly defining requirements for high-quality document sets, and train a reranker called RubricRanker using a two-stage framework. RubricRanker improves performance by 2.6 points on four deep research benchmarks and generalizes to five RAG benchmarks.
Retrieval systems help deep research agents generate high-quality answers by providing relevant documents. However, existing retrievers typically select documents through relevance matching, while individually well-matched top-$k$ documents may not form a \textit{set} that satisfies the complex information needs of an agent query (\eg, diverse, concise and authoritative documents). In this paper, we propose search-oriented rubrics that \textit{explicitly} define the requirements that high-quality document sets should satisfy for each agent query. Our search rubrics are organized into a hierarchical structure and synthesized using a powerful LLM. Based on these search rubrics, we further train a document reranker \textbf{RubricRanker} to select a high-quality subset from retrieved documents. We design a two-stage training framework that consists of rubrics-guided supervised fine-tuning and rubric-based reinforcement learning. Extensive experiments demonstrate that RubricRanker outperforms the strongest baseline by 2.6 points on four deep research benchmarks and generalizes well to five RAG benchmarks.