Does Context Matter? ContextualJudgeBench for Evaluating LLM-based Judges in Contextual Settings
This addresses a gap in evaluating LLM-based judges for contextual settings, which is crucial for practitioners in AI development and monitoring, though it is incremental as it focuses on benchmarking rather than a new method.
The paper tackles the problem that LLM-based judges are typically evaluated only on non-contextual scenarios, despite the prevalence of contextual settings like RAG and summarization, by proposing ContextualJudgeBench, a benchmark with 2,000 response pairs across eight splits, and finds that even state-of-the-art models like OpenAI's o1 achieve only 55% consistent accuracy in these challenging evaluations.
The large language model (LLM)-as-judge paradigm has been used to meet the demand for a cheap, reliable, and fast evaluation of model outputs during AI system development and post-deployment monitoring. While judge models -- LLMs finetuned to specialize in assessing and critiquing model outputs -- have been touted as general purpose evaluators, they are typically evaluated only on non-contextual scenarios, such as instruction following. The omission of contextual settings -- those where external information is used as context to generate an output -- is surprising given the increasing prevalence of retrieval-augmented generation (RAG) and summarization use cases. Contextual assessment is uniquely challenging, as evaluation often depends on practitioner priorities, leading to conditional evaluation criteria (e.g., comparing responses based on factuality and then considering completeness if they are equally factual). To address the gap, we propose ContextualJudgeBench, a judge benchmark with 2,000 challenging response pairs across eight splits inspired by real-world contextual evaluation scenarios. We build our benchmark with a multi-pronged data construction pipeline that leverages both existing human annotations and model-based perturbations. Our comprehensive study across 11 judge models and 9 general purpose models, reveals that the contextual information and its assessment criteria present a significant challenge to even state-of-the-art models. For example, OpenAI's o1, the best-performing model, barely reaches 55% consistent accuracy.