BiCAA: Bidirectional Credit Assignment for Search-Augmented Agent
For researchers working on search-augmented agents and reinforcement learning, BiCAA offers a novel reward shaping method that improves training stability and efficiency, though the performance gains are competitive rather than state-of-the-art.
BiCAA addresses the sparse reward problem in multi-step search-augmented agents by introducing a bidirectional credit assignment framework that combines forward solvability gain and hindsight success criticality to provide dense process rewards. Experiments show it stabilizes policy optimization, reduces redundant search, and achieves competitive performance on search-augmented QA benchmarks.
Multi-step search is a fundamental capability for search agents, enabling them to iteratively acquire, refine, and integrate external evidence for complex reasoning QA. However, vanilla GRPO allocates rewards exclusively based on the model's final outputs, yielding outcome-only supervision with no supervisory signals for intermediate reasoning steps. Such sparse supervision easily causes training instability and redundant search behaviors on multi-step search tasks. To mitigate this limitation, we adopt process reward to deliver stepwise supervision signals. For this process reward, we propose two complementary criteria to judge each search step: whether the step yields new evidence to facilitate problem solving, and whether it forms an efficient, pivotal intermediate decision within the overall reasoning trajectory. Building on this insight, we propose BiCAA: a bidirectional credit assignment framework that delivers dense, distinguishing process rewards for search-augmented agents. BiCAA builds bidirectional process rewards by fusing two complementary signals: forward solvability gain and hindsight success criticality. The former quantifies step-wise improvements in answer plausibility, while the latter evaluates each step's necessity for final success via hindsight outcome-based criticality scoring. We modulate and aggregate the two signals and then fuse them with the outcome reward. Experiments on search-augmented QA benchmarks show that BiCAA stabilizes policy optimization, reduces redundant search behavior, and achieves competitive performance.