ARCO: Adaptive Rubric with Co-Evolution for Multi-Step LLM-Based Agents
For researchers training multi-step LLM agents, ARCO provides a method for interpretable, step-level credit assignment without step-level labels, improving performance and diagnostic capability.
ARCO introduces a rubric-based reward framework for multi-step LLM agents that assigns step-level rewards via a shared backbone with generation and score heads, enabling co-evolution of rubric and policy. It achieves state-of-the-art Exact Match on HotpotQA, 2WikiMultiHopQA, and MuSiQue, outperforming outcome-, rubric-, and process-reward baselines in every setting.
Reinforcement learning for multi-step LLM agents often relies on scalar rewards that indicate success but cannot explain why a trajectory is good or bad. Rubric-based rewards improve interpretability through natural-language criteria, but existing methods score at the trajectory level and freeze the scorer behind a closed-source judge, leaving step-level credit assignment unresolved and the judge itself static. We propose ARCO (Adaptive Rubric CO-evolution), a rubric framework in which a same-scale model $μ$ shares a backbone with two heads: a generation head that produces per-step criteria, and a score head that predicts rubric-conditioned step-level rewards. A trajectory decomposition constraint ties the sum of step rewards to the terminal outcome, enabling credit assignment without step-level labels, while $μ$ and the policy $π$ are jointly updated on on-policy data so that the rubric content and the scoring function co-evolve at the parameter level. Across HotpotQA, 2WikiMultiHopQA, and MuSiQue with two open-source backbones, ARCO improves the best EM in every setting over strong outcome-, rubric-, and process-reward baselines, and analyses show that its rubrics are step-specific, robust to design choices, and useful for diagnosing agent behavior. Codes and data are available at https://github.com/zihangtian/ARCO.