Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERT
This provides a more efficient evaluation method for dialogue systems, but it is incremental as it applies an existing model (BERT) to a new task.
The paper tackles the problem of automatically evaluating the naturalness of generated language in task-oriented dialogues, which previously required costly human labor, and shows that fine-tuning BERT outperforms baselines like support vector machines, bi-directional LSTMs, and BLEURT.
This paper presents an automatic method to evaluate the naturalness of natural language generation in dialogue systems. While this task was previously rendered through expensive and time-consuming human labor, we present this novel task of automatic naturalness evaluation of generated language. By fine-tuning the BERT model, our proposed naturalness evaluation method shows robust results and outperforms the baselines: support vector machines, bi-directional LSTMs, and BLEURT. In addition, the training speed and evaluation performance of naturalness model are improved by transfer learning from quality and informativeness linguistic knowledge.