When LLM Judge Scores Look Good but Best-of-N Decisions Fail
This addresses a critical issue for practitioners deploying LLMs in selection tasks, revealing that standard validation metrics are insufficient and offering an incremental improvement through better auditing methods.
The paper tackles the problem that using LLMs as judges with global correlation metrics can be misleading for best-of-n selection tasks, showing that a judge with moderate global correlation (r=0.47) captures only 21.0% of potential improvement over random choice in a benchmark, while explicit pairwise judging recovers much of this lost signal, raising recovery to 61.2%.
Large language models are often used as judges to score candidate responses, then validated with a single global metric such as correlation with reference labels. This can be misleading when the real deployment task is best-of-n selection within a prompt. In a 5,000-prompt best-of-4 benchmark from Chatbot Arena, a judge with moderate global correlation (r = 0.47) captures only 21.0% of the improvement that perfect selection would achieve over random choice. The gap arises because global agreement is driven largely by prompt-level baseline effects, while selection depends on within-prompt ranking: within-prompt correlation is only r_within = 0.27, and coarse pointwise scoring creates ties in 67% of pairwise comparisons. In a matched-pair best-of-2 audit, explicit pairwise judging recovers much of this lost signal, raising recovery from 21.1% to 61.2%. For judge-based selection, the relevant audit should report within-prompt signal, tie rates, and recovery/top-1 accuracy, not global agreement alone.