AdaRubric: Task-Adaptive Rubrics for LLM Agent Evaluation
This addresses the challenge of reliable evaluation for LLM agents in tasks like web navigation and code debugging, offering a novel approach that eliminates the need for manual rubric engineering.
The paper tackles the problem of evaluating LLM agents by introducing AdaRubric, a method that generates task-specific evaluation rubrics from task descriptions, achieving a Pearson correlation of 0.79 with human judgments and improving task success rates by up to 8.5 percentage points over baselines.
LLM-as-Judge evaluation fails agent tasks because a fixed rubric cannot capture what matters for this task: code debugging demands Correctness and Error Handling; web navigation demands Goal Alignment and Action Efficiency. We present ADARUBRIC, which closes this gap by generating task-specific evaluation rubrics on the fly from task descriptions, scoring trajectories step-by-step with confidence-weighted per-dimension feedback, and filtering preference pairs with the novel DimensionAwareFilter - a provably necessary condition for preventing high-scoring dimensions from masking dimension-level failures. On WebArena and ToolBench, ADARUBRIC achieves Pearson r=0.79 human correlation (+0.16 over the best static baseline) with deployment-grade reliability (Krippendorff's $α$=0.83). DPO agents trained on ADARUBRIC preference pairs gain +6.8 to +8.5 pp task success over Prometheus across three benchmarks; gains transfer to SWE-bench code repair (+4.9 pp) and accelerate PPO convergence by +6.6 pp at 5K steps - both without any rubric engineering. Code: https://github.com/alphadl/AdaRubrics.