Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
This addresses the need for more accurate and scalable evaluation in AI, particularly for tasks requiring complex verification, though it is incremental in applying tool-integrated reasoning to judge training.
The paper tackled the problem of LLM judges being limited to text-based reasoning by proposing TIR-Judge, an RL framework that integrates a code executor for precise evaluation, resulting in performance gains of up to 6.4% and 7.7% over strong baselines on benchmarks.
Large Language Models (LLMs) are widely used as judges to evaluate response quality, providing a scalable alternative to human evaluation. However, most LLM judges operate solely on intrinsic text-based reasoning, limiting their ability to verify complex constraints or perform accurate computation. Motivated by the success of tool-integrated reasoning (TIR) in numerous tasks, we propose TIR-Judge, an end-to-end RL framework for training LLM judges that integrates a code executor for precise evaluation. TIR-Judge is built on three principles: (i) diverse training across verifiable and non-verifiable domains, (ii) flexible judgment formats (pointwise, pairwise, listwise), and (iii) iterative RL that bootstraps directly from the initial model without distillation. On seven public benchmarks, TIR-Judge surpasses strong reasoning-based judges by up to 6.4% (pointwise) and 7.7% (pairwise), and achieves listwise performance comparable to Claude-Opus-4 despite having only 8B parameters. Remarkably, TIR-Judge-Zero - trained entirely without distilled judge trajectories, matches the performance of distilled variants, demonstrating that tool-augmented judges can self-evolve through iterative reinforcement learning.