CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic
This addresses inefficiencies in training search agents for multi-hop reasoning tasks, offering a more stable and effective method, though it appears incremental as it builds on existing tool-integrated reasoning approaches.
The paper tackles the problem of sparse rewards in search agents for complex question-answering by introducing CriticSearch, a fine-grained credit-assignment framework that uses a retrospective critic to provide dense, turn-level feedback, resulting in faster convergence, improved training stability, and higher performance across diverse benchmarks.
Tool-Integrated Reasoning (TIR) with search engines enables large language models to iteratively retrieve up-to-date external knowledge, enhancing adaptability and generalization in complex question-answering tasks. However, existing search agent pipelines typically depend on reinforcement learning based optimization, which often suffers from sparse outcome rewards, leading to inefficient exploration and unstable training. We introduce CriticSearch, a fine-grained credit-assignment framework that supplies dense, turn-level feedback via a retrospective critic mechanism. During training, a frozen, asymmetric critique LLM retrospectively evaluates each turn using privileged information from the full trajectory and gold answers, converting these assessments into stable, dense rewards that guide policy improvement. Experimental results across diverse multi-hop reasoning benchmarks demonstrate that CriticSearch consistently outperforms existing baselines, achieving faster convergence, improved training stability, and higher performance.