Pragmatically Appropriate Diversity for Dialogue Evaluation
This work addresses the challenge of assessing diverse responses in neural dialogue agents for conversational AI, but it is incremental as it builds on existing diversity metrics by incorporating pragmatic constraints.
The paper tackled the problem of evaluating dialogue diversity by introducing Pragmatically Appropriate Diversity, which considers how speech acts constrain response options, and found that speech acts significantly predict the diversity of possible next responses in a human-created dataset.
Linguistic pragmatics state that a conversation's underlying speech acts can constrain the type of response which is appropriate at each turn in the conversation. When generating dialogue responses, neural dialogue agents struggle to produce diverse responses. Currently, dialogue diversity is assessed using automatic metrics, but the underlying speech acts do not inform these metrics. To remedy this, we propose the notion of Pragmatically Appropriate Diversity, defined as the extent to which a conversation creates and constrains the creation of multiple diverse responses. Using a human-created multi-response dataset, we find significant support for the hypothesis that speech acts provide a signal for the diversity of the set of next responses. Building on this result, we propose a new human evaluation task where creative writers predict the extent to which conversations inspire the creation of multiple diverse responses. Our studies find that writers' judgments align with the Pragmatically Appropriate Diversity of conversations. Our work suggests that expectations for diversity metric scores should vary depending on the speech act.