Generating and Refining Dynamic Evaluation Rubrics for LLM-as-a-Judge
This work addresses the problem of reducing reliance on human annotation for creating evaluation rubrics, which is significant for researchers and practitioners using LLM-as-a-Judge to evaluate other LLMs.
This paper introduces a training-free method to automatically generate fine-grained evaluation rubrics for LLM-as-a-Judge, achieving competitive performance across four benchmarks. Additionally, it presents an iterative fine-tuning method for rubric generators, where a fine-tuned 14B model outperforms larger proprietary models in rubric generation.
LLM-as-a-Judge is a scalable alternative to human evaluation, yet existing rubric-based methods rely on human-annotated data such as reference answers or expert-crafted rubrics. We propose to automatically generate fine-grained evaluation rubrics without any human annotation. Our training-free method generates rubrics at dataset-specific and instance-specific granularities, achieving performance competitive with existing methods across four benchmarks. We further present a method that iteratively fine-tunes a rubric generator model via meta-judge reward signals. The fine-tuned generator outperforms all existing baselines in both pairwise and pointwise evaluation. Notably, a fine-tuned 14B rubric generator outperforms a much larger proprietary model at rubric generation, showing the effectiveness of our fine-tuning strategy.