Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability
This addresses the need for flexible, interpretable, and reference-free evaluation in NLG, offering a potential new paradigm for researchers and practitioners, though it builds incrementally on existing LLM-based evaluation methods.
The paper tackles the problem of evaluating natural language generation (NLG) tasks by proposing Themis, an LLM trained on a new large-scale corpus (NLG-Eval) with human and GPT-4 annotations, which achieves superior performance on various NLG tasks without needing references and generalizes to unseen tasks, surpassing GPT-4 and other models.
The evaluation of natural language generation (NLG) tasks is a significant and longstanding research area. With the recent emergence of powerful large language models (LLMs), some studies have turned to LLM-based automatic evaluation methods, which demonstrate great potential to become a new evaluation paradigm following traditional string-based and model-based metrics. However, despite the improved performance of existing methods, they still possess some deficiencies, such as dependency on references and limited evaluation flexibility. Therefore, in this paper, we meticulously construct a large-scale NLG evaluation corpus NLG-Eval with annotations from both human and GPT-4 to alleviate the lack of relevant data in this field. Furthermore, we propose Themis, an LLM dedicated to NLG evaluation, which has been trained with our designed multi-perspective consistency verification and rating-oriented preference alignment methods. Themis can conduct flexible and interpretable evaluations without references, and it exhibits superior evaluation performance on various NLG tasks, simultaneously generalizing well to unseen tasks and surpassing other evaluation models, including GPT-4.