On the Effectiveness of Automated Metrics for Text Generation Systems
This work addresses the problem of unreliable evaluation in text generation for researchers and practitioners, though it is incremental as it builds on existing evaluation methods.
The paper tackles the challenge of evaluating text generation systems by proposing a theoretical framework that accounts for uncertainties like imperfect automated metrics and small test sets, and demonstrates its application on WMT 21 and Spot-The-Bot data to improve evaluation reliability.
A major challenge in the field of Text Generation is evaluation because we lack a sound theory that can be leveraged to extract guidelines for evaluation campaigns. In this work, we propose a first step towards such a theory that incorporates different sources of uncertainty, such as imperfect automated metrics and insufficiently sized test sets. The theory has practical applications, such as determining the number of samples needed to reliably distinguish the performance of a set of Text Generation systems in a given setting. We showcase the application of the theory on the WMT 21 and Spot-The-Bot evaluation data and outline how it can be leveraged to improve the evaluation protocol regarding the reliability, robustness, and significance of the evaluation outcome.