RIFT: A RubrIc Failure Mode Taxonomy and Automated Diagnostics
This addresses the need for principled rubric diagnostics in LLM evaluation, though it is incremental as it builds on existing rubric-based methods.
The paper tackled the problem of diagnosing rubric quality issues in LLM benchmarks by introducing RIFT, a taxonomy for characterizing failure modes, and achieved fair human agreement (87% pairwise agreement, 0.64 Cohen's kappa) and automated metrics with up to 0.86 F1 score.
Rubric-based evaluation is widely used in LLM benchmarks and training pipelines for open-ended, less verifiable tasks. While prior work has demonstrated the effectiveness of rubrics using downstream signals such as reinforcement learning outcomes, there remains no principled way to diagnose rubric quality issues from such aggregated or downstream signals alone. To address this gap, we introduce RIFT: RubrIc Failure mode Taxonomy, a taxonomy for systematically characterizing failure modes in rubric composition and design. RIFT consists of eight failure modes organized into three high-level categories: Reliability Failures, Content Validity Failures, and Consequential Validity Failures. RIFT is developed using grounded theory by iteratively annotating rubrics drawn from five diverse benchmarks spanning general instruction following, code generation, creative writing, and expert-level deep research, until no new failure modes are identified. We evaluate the consistency of the taxonomy by measuring agreement among independent human annotators, observing fair agreement overall (87% pairwise agreement and 0.64 average Cohen's kappa). Finally, to support scalable diagnosis, we propose automated rubric quality metrics and show that they align with human failure-mode annotations, achieving up to 0.86 F1.