Spurious Correlations in Reference-Free Evaluation of Text Generation
This addresses a critical problem for researchers and practitioners in natural language generation by exposing flaws in widely used evaluation methods, though it is incremental as it builds on existing metric design approaches.
The paper identified that reference-free evaluation metrics for text summarization and dialog generation rely on spurious correlations with features like word overlap and length, leading to high error rates in ranking state-of-the-art systems, and showed that designing metrics to avoid these features can mitigate the errors.
Model-based, reference-free evaluation metrics have been proposed as a fast and cost-effective approach to evaluate Natural Language Generation (NLG) systems. Despite promising recent results, we find evidence that reference-free evaluation metrics of summarization and dialog generation may be relying on spurious correlations with measures such as word overlap, perplexity, and length. We further observe that for text summarization, these metrics have high error rates when ranking current state-of-the-art abstractive summarization systems. We demonstrate that these errors can be mitigated by explicitly designing evaluation metrics to avoid spurious features in reference-free evaluation.