JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks
This addresses the evaluation dilemma between rigor and flexibility for AI agents in professional domains like business and medicine, representing a novel method for a known bottleneck.
The paper tackles the problem of evaluating AI agents on open-ended professional tasks by proposing JADE, a two-layer framework that combines expert-grounded evaluation skills with dynamic claim-level assessment, improving stability and revealing critical failure modes missed by LLM-based evaluators on BizBench.
Evaluating agentic AI on open-ended professional tasks faces a fundamental dilemma between rigor and flexibility. Static rubrics provide rigorous, reproducible assessment but fail to accommodate diverse valid response strategies, while LLM-as-a-judge approaches adapt to individual responses yet suffer from instability and bias. Human experts address this dilemma by combining domain-grounded principles with dynamic, claim-level assessment. Inspired by this process, we propose JADE, a two-layer evaluation framework. Layer 1 encodes expert knowledge as a predefined set of evaluation skills, providing stable evaluation criteria. Layer 2 performs report-specific, claim-level evaluation to flexibly assess diverse reasoning strategies, with evidence-dependency gating to invalidate conclusions built on refuted claims. Experiments on BizBench show that JADE improves evaluation stability and reveals critical agent failure modes missed by holistic LLM-based evaluators. We further demonstrate strong alignment with expert-authored rubrics and effective transfer to a medical-domain benchmark, validating JADE across professional domains. Our code is publicly available at https://github.com/smiling-world/JADE.