Referenceless Quality Estimation for Natural Language Generation
This addresses the need for cost-effective evaluation in natural language generation, though it is incremental as it builds on existing quality estimation approaches.
The paper tackles the problem of evaluating natural language generation without human references by proposing a referenceless quality estimation method using recurrent neural networks, achieving results comparable to similar tasks and showing a 21% improvement in correlation with synthetic data.
Traditional automatic evaluation measures for natural language generation (NLG) use costly human-authored references to estimate the quality of a system output. In this paper, we propose a referenceless quality estimation (QE) approach based on recurrent neural networks, which predicts a quality score for a NLG system output by comparing it to the source meaning representation only. Our method outperforms traditional metrics and a constant baseline in most respects; we also show that synthetic data helps to increase correlation results by 21% compared to the base system. Our results are comparable to results obtained in similar QE tasks despite the more challenging setting.