J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
This addresses the bottleneck in AI progress due to evaluation quality, offering a method to enhance LLM judges for more reliable AI assessment, though it appears incremental as it builds on existing LLM-as-a-Judge approaches.
The paper tackles the problem of improving LLM-as-a-Judge models by optimizing their chain-of-thought reasoning, introducing J1, a reinforcement learning framework that achieves state-of-the-art performance across multiple benchmarks, with J1-Qwen-32B outperforming larger models like o1-mini, o3, and DeepSeek-R1 on some benchmarks.
The progress of AI is bottlenecked by the quality of evaluation, making powerful LLM-as-a-Judge models a core solution. The efficacy of these judges depends on their chain-of-thought reasoning, creating a critical need for methods that can effectively optimize this reasoning process. In this work, we introduce J1, a reinforcement learning framework for teaching LLM judges to think before making decisions. Our core contribution lies in converting all judgment tasks for non-verifiable and verifiable prompts into a unified format with verifiable rewards, enabling direct optimization of evaluation quality while mitigating positional bias. We then use RL to train thinking-judges at scales of 8B, 32B, and 70B and show that they obtain state-of-the-art performance across multiple benchmarks. In particular, J1-Qwen-32B, our multitasked pointwise and pairwise judge also outperforms o1-mini, o3, and a much larger 671B DeepSeek-R1 on some benchmarks, while only training on synthetic data. Through comprehensive ablations of pairwise, pointwise, and multitask J1 variants, we demonstrate the effectiveness of our approach across seed prompts, reward strategies, and training recipes. Qualitative analysis reveals that J1 develops systematic evaluation strategies, including dynamic criteria generation, reference answer creation, iterative self-correction of initial assessments, and feedback generation for low-quality responses.