Fusion-Eval: Integrating Assistant Evaluators with LLMs
This addresses the problem of natural language system evaluation for researchers and practitioners, representing a strong incremental improvement over baseline methods.
The paper tackles the challenge of evaluating natural language systems by introducing Fusion-Eval, an approach that integrates insights from multiple assistant evaluators using LLMs, achieving a 0.962 system-level Kendall-Tau correlation with humans on SummEval and a 0.744 turn-level Spearman correlation on TopicalChat.
Evaluating natural language systems poses significant challenges, particularly in the realms of natural language understanding and high-level reasoning. In this paper, we introduce 'Fusion-Eval', an innovative approach that leverages Large Language Models (LLMs) to integrate insights from various assistant evaluators. The LLM is given the example to evaluate along with scores from the assistant evaluators. Each of these evaluators specializes in assessing distinct aspects of responses. Fusion-Eval achieves a 0.962 system-level Kendall-Tau correlation with humans on SummEval and a 0.744 turn-level Spearman correlation on TopicalChat, which is significantly higher than baseline methods. These results highlight Fusion-Eval's significant potential in the realm of natural language system evaluation.