DualEval: Joint Model-Item Calibration for Unified LLM Evaluation
For LLM evaluation practitioners, DualEval provides a unified framework that integrates two previously disconnected evaluation signals, enabling more sample-efficient and interpretable evaluation pipelines.
DualEval introduces a latent model-item calibration framework that jointly estimates model ability, item difficulty, and sharpness, unifying static benchmarks and arena-style preference data. Applied across four domains with 18 frontier LLMs, it produces reliable model rankings and enables benchmark compression and anomaly detection.
Current LLM evaluation relies on two complementary but often disconnected signals: static benchmarks with objective correctness labels and arena-style preference data that better reflect open-ended user interactions. We introduce DualEval, a latent model-item calibration framework that represents models and evaluation items in a shared space, jointly estimating model ability together with item difficulty and sharpness. We apply DualEval across four domains: coding, math, miscellaneous domain-knowledge tasks, and generic everyday user queries. Our evaluation uses 18 frontier LLMs, static benchmark labels, and reward-model scores validated against held-out human preferences for open-ended model responses. Empirically, our framework produces reliable and balanced model rankings, and its learned item-level profiles support downstream applications such as benchmark compression for sample-efficient evaluation and anomaly detection for contamination or outlier analysis. Overall, DualEval unifies static and arena-style evaluation through joint model-item calibration, producing model rankings and item-level diagnostics that support more sample-efficient, interpretable, and auditable evaluation pipelines.