Self-Supervised Contrastive Learning for Efficient User Satisfaction Prediction in Conversational Agents
This work addresses the problem of high annotation costs and slow turnaround times for developing conversational agents in large-scale commercial systems, though it is incremental as it builds on existing self-supervised and transfer learning techniques.
The paper tackles the challenge of predicting user satisfaction in conversational agents with limited labeled data by proposing a self-supervised contrastive learning approach that leverages unlabeled data and a few-shot transfer method, resulting in significantly reduced annotation needs and improved generalization on unseen skills.
Turn-level user satisfaction is one of the most important performance metrics for conversational agents. It can be used to monitor the agent's performance and provide insights about defective user experiences. Moreover, a powerful satisfaction model can be used as an objective function that a conversational agent continuously optimizes for. While end-to-end deep learning has shown promising results, having access to a large number of reliable annotated samples required by these methods remains challenging. In a large-scale conversational system, there is a growing number of newly developed skills, making the traditional data collection, annotation, and modeling process impractical due to the required annotation costs as well as the turnaround times. In this paper, we suggest a self-supervised contrastive learning approach that leverages the pool of unlabeled data to learn user-agent interactions. We show that the pre-trained models using the self-supervised objective are transferable to the user satisfaction prediction. In addition, we propose a novel few-shot transfer learning approach that ensures better transferability for very small sample sizes. The suggested few-shot method does not require any inner loop optimization process and is scalable to very large datasets and complex models. Based on our experiments using real-world data from a large-scale commercial system, the suggested approach is able to significantly reduce the required number of annotations, while improving the generalization on unseen out-of-domain skills.