JudgeLRM: Large Reasoning Models as a Judge
This addresses the need for scalable and accurate AI evaluators in reasoning-intensive tasks, offering a novel approach that improves performance over existing methods.
The paper tackles the problem of using large language models as evaluators in domains requiring complex reasoning, where supervised fine-tuning falls short, and introduces JudgeLRM, a family of judgment-oriented models trained with reinforcement learning that outperform baselines and state-of-the-art models, such as exceeding GPT-4 with JudgeLRM-3B/4B and outperforming DeepSeek-R1 by over 2% in F1 score.
Large Language Models (LLMs) are increasingly adopted as evaluators, offering a scalable alternative to human annotation. However, existing supervised fine-tuning (SFT) approaches often fall short in domains that demand complex reasoning. Judgment is inherently reasoning-intensive: beyond surface-level scoring, it requires verifying evidence, identifying errors, and justifying decisions. Through the analysis of evaluation tasks, we find a negative correlation between SFT performance gains and the proportion of reasoning-demanding samples, revealing the limits of SFT in such scenarios. To address this, we introduce JudgeLRM, a family of judgment-oriented LLMs, trained using reinforcement learning (RL) with judge-wise, outcome-driven rewards to activate reasoning capabilities. JudgeLRM consistently outperform SFT-tuned baselines in the same size, as well as other RL and SFT variants, and even surpass state-of-the-art reasoning models: notably, JudgeLRM-3B/4B exceeds GPT-4, while JudgeLRM-7B/8B/14B outperforms DeepSeek-R1 by over 2% in F1 score, with particularly strong gains on reasoning-heavy tasks. Our findings underscore the value of RL in unlocking reasoning-aligned LLM judges.