Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models
This addresses the problem of accurate AI evaluation for researchers and practitioners, though it is incremental as it builds on existing graphical model frameworks.
The paper tackles the problem of aggregating binary judgments from multiple annotators (including LLMs) when they exhibit dependencies, showing that classical methods assuming independence can produce confidently incorrect predictions. The proposed dependence-aware method based on Ising models demonstrates improved performance over baselines on three real-world datasets.
Large-scale AI evaluation increasingly relies on aggregating binary judgments from $K$ annotators, including LLMs used as judges. Most classical methods, e.g., Dawid-Skene or (weighted) majority voting, assume annotators are conditionally independent given the true label $Y\in\{0,1\}$, an assumption often violated by LLM judges due to shared data, architectures, prompts, and failure modes. Ignoring such dependencies can yield miscalibrated posteriors and even confidently incorrect predictions. We study label aggregation through a hierarchy of dependence-aware models based on Ising graphical models and latent factors. For class-dependent Ising models, the Bayes log-odds is generally quadratic in votes; for class-independent couplings, it reduces to a linear weighted vote with correlation-adjusted parameters. We present finite-$K$ examples showing that methods based on conditional independence can flip the Bayes label despite matching per-annotator marginals. We prove separation results demonstrating that these methods remain strictly suboptimal as the number of judges grows, incurring nonvanishing excess risk under latent factors. Finally, we evaluate the proposed method on three real-world datasets, demonstrating improved performance over the classical baselines.