Knockout LLM Assessment: Using Large Language Models for Evaluations through Iterative Pairwise Comparisons
This addresses the need for more reliable LLM-based evaluations in domains like education and translation, though it is incremental as it builds on existing pairwise comparison approaches.
The paper tackled the problem of LLM-as-a-Judge methods lacking a global ranking perspective by introducing Knockout Assessment, a method using iterative pairwise comparisons in a knockout tournament system, which improved scoring accuracy by increasing Pearson correlation with expert evaluations by 0.07 on average for exam scoring and machine translation tasks.
Large Language Models (LLMs) have shown to be effective evaluators across various domains such as machine translations or the scientific domain. Current LLM-as-a-Judge approaches rely mostly on individual assessments or a single round of pairwise assessments, preventing the judge LLM from developing a global ranking perspective. To address this, we present Knockout Assessment, an LLM-asa Judge method using a knockout tournament system with iterative pairwise comparisons. Experiments across three LLMs on two datasets show that knockout assessment improves scoring accuracy, increasing Pearson correlation with expert evaluations by 0.07 on average for university-level exam scoring and machine translation evaluations, aligning LLM assessments more closely with human scoring.