When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validity
This addresses a critical issue for researchers and practitioners using LLM benchmarks, as it reveals that current methods may produce misleading results, though it is incremental in offering diagnostic tools rather than a new evaluation paradigm.
The paper tackles the problem of unreliable LLM-judged benchmarks by showing that design failures lead to high noise in rankings, with unexplained variance exceeding 90% for some judges and factor correlations above 0.93, undermining validity.
LLM-judged benchmarks are increasingly used to evaluate complex model behaviors, yet their design introduces failure modes absent in conventional ground-truth based benchmarks. We argue that without tight objectives and verifiable constructions, benchmark rankings can produce high-confidence rankings that are in fact largely noise. We introduce two mechanisms to diagnose these issues. Schematic adherence quantifies how much of a judge's overall verdict is explained by the explicit evaluation schema, revealing unexplained variance when judges deviate from their own rubric. Psychometric validity aggregates internal consistency and discriminant validity signals to quantify irreducible uncertainty in any benchmarking run. Applying these tools to Arena-Hard Auto, we find severe schema incoherence and factor collapse across popular judges: for example, unexplained variance exceeding 90 percent for DeepSeek-R1-32B and factor correlations above 0.93 for most criteria. We also show that the ELO-style aggregation used by Arena-Hard Auto collapses and masks genuine ranking uncertainty. Our results highlight design failures that undermine validity and offer actionable principles for building better-scoped, reliability-aware LLM-judged benchmarks. We released our code and dataset at https://github.com/penfever/judgment-to-noise