HERALD: An Annotation Efficient Method to Detect User Disengagement in Social Conversations
This addresses the need for efficient user engagement detection in dialog systems, though it is incremental as it builds on existing methods with a focus on annotation efficiency.
The paper tackles the problem of detecting user disengagement in open-domain dialog systems by proposing HERALD, an annotation-efficient framework that uses labeling heuristics and denoising to reduce manual labeling, achieving 86% detection accuracy in two dialog corpora.
Open-domain dialog systems have a user-centric goal: to provide humans with an engaging conversation experience. User engagement is one of the most important metrics for evaluating open-domain dialog systems, and could also be used as real-time feedback to benefit dialog policy learning. Existing work on detecting user disengagement typically requires hand-labeling many dialog samples. We propose HERALD, an efficient annotation framework that reframes the training data annotation process as a denoising problem. Specifically, instead of manually labeling training samples, we first use a set of labeling heuristics to label training samples automatically. We then denoise the weakly labeled data using the Shapley algorithm. Finally, we use the denoised data to train a user engagement detector. Our experiments show that HERALD improves annotation efficiency significantly and achieves 86% user disengagement detection accuracy in two dialog corpora.