GrandJury: A Collaborative Machine Learning Model Evaluation Protocol for Dynamic Quality Rubrics
This addresses the issue for AI practitioners of aligning model evaluations with dynamic user needs rather than static benchmarks, though it is incremental in proposing a new protocol rather than a fundamental breakthrough.
The paper tackles the problem of evaluating generative machine learning models in dynamic contexts by introducing GrandJury, a collaborative protocol that uses time-decayed aggregation, traceability, and multi-rater human judgment to enable pluralistic and accountable evaluation, with an open-source implementation provided.
Generative Machine Learning models have become central to modern systems, powering applications in creative writing, summarization, multi-hop reasoning, and context-aware dialogue. These models underpin large-scale AI assistants, workflow automation, and autonomous decision-making. In such domains, acceptable response is rarely absolute or static, but plural and highly context-dependent. Yet standard evaluation regimes still rely on static, benchmark-style tests, incentivizing optimization toward leaderboard scores rather than alignment with dynamic user needs or evolving realities. GrandJury introduces a formal evaluation protocol combining time-decayed aggregation, complete traceability, with the support of dynamic, transparent task rubric attribution, and multi-rater human judgment. Together, these elements enable pluralistic, accountable evaluation that captures evolving consensus and surfaces disagreement. We provide an open-source implementation (grandjury PyPI package) and a public collection of Large Language Model (LLM) inference outputs to illustrate the need and method. GrandJury provides a new paradigm for AI practitioners when evaluating machine learning outputs without absolute ground truth.