Rubrics on Trial: Evolving Rubrics from a Single Query via Synthetic Pairwise Evidence
For practitioners needing to evaluate LLM outputs without human annotations, this provides a fully automated method to generate discriminative rubrics.
The paper introduces Rubrics on Trial, a query-only framework that evolves rubrics from scratch without external annotations or model training, using synthetic pairwise evidence to validate each rubric. It achieves the best average accuracy across five preference benchmark suites, leading on six of seven evaluation sets.
Rubrics provide structured, fine-grained signals for training and evaluating large language models (LLMs). Yet reliable query-specific rubrics are difficult to construct. Existing approaches often derive supervision from human-written rubrics, preference data, or sampled responses. Direct query-to-rubric generation avoids these resources, but provides no explicit check that a plausible rubric is useful. Such a rubric may fail to distinguish answer quality, reward an optional style, or penalize a valid alternative strategy. We introduce Rubrics on Trial, a query-only framework that evolves a rubric set from an empty set without external annotations or model training. It derives supervision solely from synthetic rubric-conditioned response pairs and validates each proposed rubric before adding it, screening out non-discriminative, over-specific, and style-only candidate rubrics. Experiments across five preference benchmark suites demonstrate the effectiveness of Rubrics on Trial, which achieves the best average accuracy and leads on six of seven evaluation sets.