What makes a good conversation? How controllable attributes affect human judgments
This work addresses the problem of improving dialogue agent quality for human users by analyzing controllable attributes, though it is incremental as it builds on existing methods for a specific task.
The study investigated how controlling attributes like repetition, specificity, response-relatedness, and question-asking affects human judgments of conversation quality in chitchat dialogue, showing that models using controllable neural text generation methods achieved clear improvements in quality.
A good conversation requires balance -- between simplicity and detail; staying on topic and changing it; asking questions and answering them. Although dialogue agents are commonly evaluated via human judgments of overall quality, the relationship between quality and these individual factors is less well-studied. In this work, we examine two controllable neural text generation methods, conditional training and weighted decoding, in order to control four important attributes for chitchat dialogue: repetition, specificity, response-relatedness and question-asking. We conduct a large-scale human evaluation to measure the effect of these control parameters on multi-turn interactive conversations on the PersonaChat task. We provide a detailed analysis of their relationship to high-level aspects of conversation, and show that by controlling combinations of these variables our models obtain clear improvements in human quality judgments.