FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge
This addresses evaluation challenges in AI for researchers and practitioners, but it is incremental as it builds on existing LLM-as-a-Judge frameworks.
The paper tackled the limitations of LLM-as-a-Judge systems, such as lack of adaptivity, biases, and inconsistency, by proposing FairJudge, which models judging as a learnable policy and uses a curriculum training approach, resulting in improved agreement and F1 scores, reduced biases, and outperforming larger models on benchmarks.
Existing LLM-as-a-Judge systems suffer from three fundamental limitations: limited adaptivity to task- and domain-specific evaluation criteria, systematic biases driven by non-semantic cues such as position, length, format, and model provenance, and evaluation inconsistency that leads to contradictory judgments across different evaluation modes (e.g., pointwise versus pairwise). To address these issues, we propose FairJudge, an adaptive, debiased, and consistent LLM-as-a-Judge. Unlike prior approaches that treat the judge as a static evaluator, FairJudge models judging behavior itself as a learnable and regularized policy. From a data-centric perspective, we construct a high-information-density judging dataset that explicitly injects supervision signals aligned with evaluation behavior. Building on this dataset, we adopt a curriculum-style SFT-DPO-GRPO training paradigm that progressively aligns rubric adherence, bias mitigation, and cross-mode consistency, while avoiding catastrophic forgetting. Experimental results on multiple internal and public benchmarks show that FairJudge consistently improves agreement and F1, reduces non-semantic biases, and outperforms substantially larger instruction-tuned LLMs. All resources will be publicly released after acceptance to facilitate future research.