Speaker Sensitive Response Evaluation Model
This addresses the problem of over-reliance on ground truth in dialogue evaluation for researchers and developers, though it is incremental as it builds on context similarity ideas.
The paper tackles the challenge of automatically evaluating open-domain dialogue responses by proposing a model that considers speaker-sensitive conversational context similarity, trained on an unlabeled Twitter corpus. It outperforms existing metrics with high correlation to human scores and generalizes to movie dialogues without retraining.
Automatic evaluation of open-domain dialogue response generation is very challenging because there are many appropriate responses for a given context. Existing evaluation models merely compare the generated response with the ground truth response and rate many of the appropriate responses as inappropriate if they deviate from the ground truth. One approach to resolve this problem is to consider the similarity of the generated response with the conversational context. In this paper, we propose an automatic evaluation model based on that idea and learn the model parameters from an unlabeled conversation corpus. Our approach considers the speakers in defining the different levels of similar context. We use a Twitter conversation corpus that contains many speakers and conversations to test our evaluation model. Experiments show that our model outperforms the other existing evaluation metrics in terms of high correlation with human annotation scores. We also show that our model trained on Twitter can be applied to movie dialogues without any additional training. We provide our code and the learned parameters so that they can be used for automatic evaluation of dialogue response generation models.