Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis
This addresses the need for better evaluation metrics in NLG tasks, offering a novel unsupervised approach that is competitive with supervised methods, though it is incremental in improving existing metric techniques.
The paper tackled the problem of building a general automatic evaluation metric for natural language generation without requiring human annotation, by introducing SESCORE, which outperformed all prior unsupervised metrics on tasks like machine translation and achieved an average Kendall correlation improvement from 0.154 to 0.195 on WMT datasets.
Is it possible to build a general and automatic natural language generation (NLG) evaluation metric? Existing learned metrics either perform unsatisfactorily or are restricted to tasks where large human rating data is already available. We introduce SESCORE, a model-based metric that is highly correlated with human judgements without requiring human annotation, by utilizing a novel, iterative error synthesis and severity scoring pipeline. This pipeline applies a series of plausible errors to raw text and assigns severity labels by simulating human judgements with entailment. We evaluate SESCORE against existing metrics by comparing how their scores correlate with human ratings. SESCORE outperforms all prior unsupervised metrics on multiple diverse NLG tasks including machine translation, image captioning, and WebNLG text generation. For WMT 20/21 En-De and Zh-En, SESCORE improve the average Kendall correlation with human judgement from 0.154 to 0.195. SESCORE even achieves comparable performance to the best supervised metric COMET, despite receiving no human-annotated training data.