On the Blind Spots of Model-Based Evaluation Metrics for Text Generation
This work addresses the reliability of evaluation metrics for text generation, which is crucial for researchers and practitioners in natural language processing, though it is incremental as it builds on existing methodologies to identify and mitigate blind spots.
The paper tackles the problem of evaluating text generation metrics by using synthetic stress tests to reveal insensitivities and biases, such as BERTScore's confusion by truncation errors and MAUVE's insensitivity to errors in specific positions, and suggests practical workarounds for more reliable evaluation.
In this work, we explore a useful but often neglected methodology for robustness analysis of text generation evaluation metrics: stress tests with synthetic data. Basically, we design and synthesize a wide range of potential errors and check whether they result in a commensurate drop in the metric scores. We examine a range of recently proposed evaluation metrics based on pretrained language models, for the tasks of open-ended generation, translation, and summarization. Our experiments reveal interesting insensitivities, biases, or even loopholes in existing metrics. For example, we find that BERTScore is confused by truncation errors in summarization, and MAUVE (built on top of GPT-2) is insensitive to errors at the beginning or middle of generations. Further, we investigate the reasons behind these blind spots and suggest practical workarounds for a more reliable evaluation of text generation. We have released our code and data at https://github.com/cloudygoose/blindspot_nlg.