CLEV: LLM-Based Evaluation Through Lightweight Efficient Voting for Free-Form Question-Answering
This addresses the problem of efficient and reliable evaluation for researchers and practitioners in natural language processing, though it is incremental as it builds on existing LLM-based evaluation methods.
The paper tackles the challenge of evaluating free-form Question Answering by proposing CLEV, a method that uses Large Language Models as judges with lightweight voting to reduce computational costs while maintaining reliability, demonstrating consistent and scalable assessments in experiments.
Evaluating free-form Question Answering (QA) remains a challenge due to its diverse and open-ended nature. Traditional automatic metrics fail to capture semantic equivalence or accommodate the variability of open-ended responses. Leveraging Large Language Models (LLMs) as evaluators offers a promising alternative due to their strong language understanding and instruction-following capabilities. We propose Consensus via Lightweight Efficient Voting (CLEV), which employs two primary LLMs as judges and invokes a third judge only in cases of disagreement. This approach prioritizes evaluation reliability while reducing unnecessary computational demands. Through experiments, including human evaluation, we demonstrate CLEV's ability to provide consistent, scalable, and resource-efficient assessments, establishing it as a robust framework for evaluating LLMs on free-form QA.