Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QA
This addresses the need for robust evaluation methods in AI for free-form QA, though it is incremental as it builds on existing LLM-as-judge approaches.
The paper tackles the problem of evaluating open-ended question-answering tasks by proposing a reference-guided verdict method that uses multiple LLMs as judges, showing improved reliability and accuracy with a strong correlation to human evaluations.
The emergence of Large Language Models (LLMs) as chat assistants capable of generating human-like conversations has amplified the need for robust evaluation methods, particularly for open-ended tasks. Conventional metrics such as EM and F1, while useful, are inadequate for capturing the full semantics and contextual depth of such generative outputs. We propose a reference-guided verdict method that automates the evaluation process by leveraging multiple LLMs as judges. Through experiments on free-form question-answering tasks, we demonstrate that combining multiple models improves the reliability and accuracy of evaluations, especially in tasks where a single model may struggle. The results indicate a strong correlation with human evaluations, establishing the proposed method as a reliable alternative to traditional metrics.