Beyond Pointwise Scores: Decomposed Criteria-Based Evaluation of LLM Responses
This addresses the problem of nuanced evaluation in expert domains for researchers and practitioners, representing a strong specific gain rather than an incremental improvement.
The paper tackles the challenge of evaluating long-form LLM responses in high-stakes domains like law or medicine by introducing DeCE, a decomposed evaluation framework that separates precision and recall using automatically extracted criteria. DeCE achieved a correlation of r=0.78 with expert judgments, significantly outperforming traditional metrics and other LLM-based evaluators.
Evaluating long-form answers in high-stakes domains such as law or medicine remains a fundamental challenge. Standard metrics like BLEU and ROUGE fail to capture semantic correctness, and current LLM-based evaluators often reduce nuanced aspects of answer quality into a single undifferentiated score. We introduce DeCE, a decomposed LLM evaluation framework that separates precision (factual accuracy and relevance) and recall (coverage of required concepts), using instance-specific criteria automatically extracted from gold answer requirements. DeCE is model-agnostic and domain-general, requiring no predefined taxonomies or handcrafted rubrics. We instantiate DeCE to evaluate different LLMs on a real-world legal QA task involving multi-jurisdictional reasoning and citation grounding. DeCE achieves substantially stronger correlation with expert judgments ($r=0.78$), compared to traditional metrics ($r=0.12$), pointwise LLM scoring ($r=0.35$), and modern multidimensional evaluators ($r=0.48$). It also reveals interpretable trade-offs: generalist models favor recall, while specialized models favor precision. Importantly, only 11.95% of LLM-generated criteria required expert revision, underscoring DeCE's scalability. DeCE offers an interpretable and actionable LLM evaluation framework in expert domains.