CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation
This addresses unreliable model rankings in LLM-based evaluation for researchers and practitioners, offering an efficient solution to a known bottleneck.
The paper tackled the problem of systematic biases in LLM-as-judge evaluations, which can distort model rankings, by introducing CyclicJudge, a round-robin assignment strategy that eliminates bias while maintaining the cost of single-judge evaluation, with empirical validation on MT-Bench supporting the theoretical predictions.
LLM-as-judge evaluation has become standard practice for open-ended model assessment; however, judges exhibit systematic biases that cannot be eliminated by increasing the number of scenarios or generations. These biases are often similar in magnitude to the model differences that benchmarks are designed to detect, resulting in unreliable rankings when single-judge evaluations are used. This work introduces a variance decomposition that partitions benchmark score variance into scenario, generation, judge, and residual components. Based on this analysis, CyclicJudge, a round-robin assignment of judges, is demonstrated to be the optimal allocation strategy. It eliminates bias precisely while requiring each judge only once per cycle, maintaining the cost of single-judge evaluation. Empirical validation on MT-Bench supports all theoretical predictions.