DEUS: A Data-driven Approach to Estimate User Satisfaction in Multi-turn Dialogues
This addresses the challenge of scalable evaluation for digital assistants, but it is incremental as it builds on existing budget-based models for dialogue systems.
The paper tackles the problem of evaluating user satisfaction in multi-turn dialogues by proposing a context-sensitive method that models user interactions using a budget consumption concept, demonstrating effectiveness through experiments on simulated and real dialogues.
Digital assistants are experiencing rapid growth due to their ability to assist users with day-to-day tasks where most dialogues are happening multi-turn. However, evaluating multi-turn dialogues remains challenging, especially at scale. We suggest a context-sensitive method to estimate the turn-level satisfaction for dialogue considering various types of user preferences. The costs of interactions between users and dialogue systems are formulated using a budget consumption concept. We assume users have an initial interaction budget for a dialogue formed based on the task complexity and that each turn has a cost. When the task is completed, or the budget has been exhausted, users quit the dialogue. We demonstrate our method's effectiveness by extensive experimentation with a simulated dialogue platform and real multi-turn dialogues.