On conducting better validation studies of automatic metrics in natural language generation evaluation
This work tackles the problem of ensuring reliable automatic evaluation in NLG for researchers and practitioners, though it is incremental as it synthesizes and refines existing validation practices rather than introducing a new metric.
The paper addresses the methodological challenges in validating automatic metrics for natural language generation (NLG) evaluation, which are crucial due to the high cost of human evaluations. It provides best practices for conducting validation studies, demonstrates their application through an analysis of the WMT'17 metrics shared task, and offers insights into future directions for NLG metrics.
Natural language generation (NLG) has received increasing attention, which has highlighted evaluation as a central methodological concern. Since human evaluations for these systems are costly, automatic metrics have broad appeal in NLG. Research in language generation often finds situations where it is appropriate to apply existing metrics or propose new ones. The application of these metrics are entirely dependent on validation studies - studies that determine a metric's correlation to human judgment. However, there are many details and considerations in conducting strong validation studies. This document is intended for those validating existing metrics or proposing new ones in the broad context of NLG: we 1) begin with a write-up of best practices in validation studies, 2) outline how to adopt these practices, 3) conduct analyses in the WMT'17 metrics shared task\footnote{Our jupyter notebook containing the analyses is available at \url{https://github.com}}, and 4) highlight promising approaches to NLG metrics 5) conclude with our opinions on the future of this area.