SE-Search: Self-Evolving Search Agent via Memory and Dense Reward
This addresses the issue of noisy search results and slow training in autonomous information-seeking systems, representing an incremental improvement over existing methods.
The paper tackles the problem of irrelevant document accumulation and sparse rewards in retrieval augmented generation (RAG) for search agents, proposing SE-Search with memory purification, atomic query training, and dense rewards, which yields a 10.8 point absolute improvement and 33.8% relative gain over baselines on question answering benchmarks.
Retrieval augmented generation (RAG) reduces hallucinations and factual errors in large language models (LLMs) by conditioning generation on retrieved external knowledge. Recent search agents further cast RAG as an autonomous, multi-turn information-seeking process. However, existing methods often accumulate irrelevant or noisy documents and rely on sparse reinforcement learning signals. We propose \textbf{S}elf-\textbf{E}volving \textbf{Search}, a Self-Evolving Search agent that improves online search behavior through three components, memory purification, atomic query training, and dense rewards. SE-Search follows a \textit{Think-Search-Memorize} strategy that retains salient evidence while filtering irrelevant content. Atomic query training promotes shorter and more diverse queries, improving evidence acquisition. Dense rewards provide fine-grained feedback that speeds training. Experiments on single-hop and multi-hop question answering benchmarks show that \texttt{SE-Search-3B} outperforms strong baselines, yielding a $10.8$ point absolute improvement and a $33.8\%$ relative gain over Search-R1.\footnote{We will make the code and model weights publicly available upon acceptance.}