Who Judges the Judge? LLM Jury-on-Demand: Building Trustworthy LLM Evaluation Systems
This addresses the need for trustworthy LLM evaluation systems for high-stakes applications, though it is incremental as it builds on existing jury-based methods with a learning-based adaptation.
The paper tackles the problem of scalable and reliable evaluation of Large Language Models (LLMs) in high-stakes domains by proposing LLM Jury-on-Demand, a dynamic framework that selects optimal juries of LLM judges based on reliability predictors, achieving significantly higher correlation with human judgment than baselines on summarization and RAG benchmarks.
As Large Language Models (LLMs) become integrated into high-stakes domains, there is a growing need for evaluation methods that are both scalable for real-time deployment and reliable for critical decision-making. While human evaluation is reliable, it is slow and costly. Single LLM judges are biased, and static juries lack adaptability. To overcome these limitations, we propose LLM Jury-on-Demand - a dynamic, learning-based framework for scalable and context-aware evaluation. Our method trains a set of reliability predictors to assess when LLM judges will agree with human experts, leveraging token distributions, embeddings, and structural input features. This enables a fully adaptive evaluation where, for each data point, an optimal jury of the most reliable judges is dynamically selected, and their scores are aggregated using their reliability as weights. Experiments on summarization and RAG benchmarks show that our dynamic jury system achieves significantly higher correlation with human judgment than both single-judge and static-jury baselines. These results highlight the promise of adaptive, learning-based juries for building scalable, more reliable and trustworthy evaluation systems for modern LLMs in high-stakes domains.