Is Reference Necessary in the Evaluation of NLG Systems? When and Where?
This work addresses the challenge of reliable evaluation in NLG for researchers and practitioners, especially in scenarios lacking human annotations, though it is incremental in clarifying metric applicability.
The study investigated when and where reference-free metrics can replace reference-based metrics for evaluating NLG systems, finding that reference-free metrics generally correlate better with human judgment and are more sensitive to language quality issues, but their effectiveness varies by task and candidate text quality.
The majority of automatic metrics for evaluating NLG systems are reference-based. However, the challenge of collecting human annotation results in a lack of reliable references in numerous application scenarios. Despite recent advancements in reference-free metrics, it has not been well understood when and where they can be used as an alternative to reference-based metrics. In this study, by employing diverse analytical approaches, we comprehensively assess the performance of both metrics across a wide range of NLG tasks, encompassing eight datasets and eight evaluation models. Based on solid experiments, the results show that reference-free metrics exhibit a higher correlation with human judgment and greater sensitivity to deficiencies in language quality. However, their effectiveness varies across tasks and is influenced by the quality of candidate texts. Therefore, it's important to assess the performance of reference-free metrics before applying them to a new task, especially when inputs are in uncommon form or when the answer space is highly variable. Our study can provide insight into the appropriate application of automatic metrics and the impact of metric choice on evaluation performance.