User Response and Sentiment Prediction for Automatic Dialogue Evaluation
This addresses the issue of poor correlation with human judgments for developers of open-domain dialogue systems, though it is incremental as it builds on existing sentiment and generation methods.
The paper tackles the problem of automatic evaluation for open-domain dialogue systems by proposing to use the sentiment of the next user utterance, showing that their model outperforms existing metrics on written and spoken datasets.
Automatic evaluation is beneficial for open-domain dialog system development. However, standard word-overlap metrics (BLEU, ROUGE) do not correlate well with human judgements of open-domain dialog systems. In this work we propose to use the sentiment of the next user utterance for turn or dialog level evaluation. Specifically we propose three methods: one that predicts the next sentiment directly, and two others that predict the next user utterance using an utterance or a feedback generator model and then classify its sentiment. Experiments show our model outperforming existing automatic evaluation metrics on both written and spoken open-domain dialogue datasets.