Automating Text Naturalness Evaluation of NLG Systems
This work addresses the need for efficient and repeatable quality assessment in NLG development, though it is incremental as it builds on existing model-based approaches.
The paper tackled the problem of automating text naturalness evaluation for NLG systems by proposing a method using human likeliness metrics and pretrained language models, finding that larger models improve evaluation accuracy.
Automatic methods and metrics that assess various quality criteria of automatically generated texts are important for developing NLG systems because they produce repeatable results and allow for a fast development cycle. We present here an attempt to automate the evaluation of text naturalness which is a very important characteristic of natural language generation methods. Instead of relying on human participants for scoring or labeling the text samples, we propose to automate the process by using a human likeliness metric we define and a discrimination procedure based on large pretrained language models with their probability distributions. We analyze the text probability fractions and observe how they are influenced by the size of the generative and discriminative models involved in the process. Based on our results, bigger generators and larger pretrained discriminators are more appropriate for a better evaluation of text naturalness. A comprehensive validation procedure with human participants is required as follow up to check how well this automatic evaluation scheme correlates with human judgments.