ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agents
For LLM-based search agents, ARBOR addresses the problem of degenerate outcome-only rewards by providing efficient process supervision without costly verifiers or inconsistent per-query rubrics.
ARBOR introduces a reusable process-reward framework that maintains a shared rubric memory across queries, providing process-level gradient even when outcome reward is uniform. It outperforms GRPO and DAPO baselines on four multi-hop QA benchmarks, improving average LLM-judge accuracy by up to 4.2 points and converting up to 42% of zero-gradient training groups into informative ones.
LLM-based search agents are trained predominantly with outcome-only reward, leaving the search process itself unsupervised. This signal degenerates on outcome-homogeneous groups where all sampled trajectories share the same correctness, yielding zero within-group advantage and no gradient. Existing process supervision either trains a costly verifier or generates per-query rubrics that are inconsistent across queries and discarded after one use. We propose ARBOR (Adaptive Rubric Buffer for Online Reward), a reusable process-reward framework that maintains a rubric memory shared across queries. Query-local drafts induced from contrastive trajectories are admitted, consolidated into cross-query common rubrics, and retired as the policy evolves. A small active subset of common rubrics scores trajectories via sparse pairwise judging, and the resulting scores are added to the base reward, providing process-level gradient even when outcome reward is uniform. ARBOR consistently outperforms GRPO and DAPO baselines on four multi-hop QA benchmarks, raising average LLM-judge accuracy by up to 4.2 points and converting up to 42% of otherwise-zero-gradient training groups into informative ones.