Analysis of Utterance Embeddings and Clustering Methods Related to Intent Induction for Task-Oriented Dialogue
This work addresses the challenge of designing task-oriented dialogue schemas without labeled data, but it is incremental as it compares existing methods rather than introducing new ones.
The paper tackled the problem of automatically inducing intents for task-oriented dialogue by investigating unsupervised clustering methods and utterance embeddings, finding that pretrained MiniLM with Agglomerative clustering significantly improved metrics like NMI, ARI, F1, accuracy, and example coverage.
The focus of this work is to investigate unsupervised approaches to overcome quintessential challenges in designing task-oriented dialog schema: assigning intent labels to each dialog turn (intent clustering) and generating a set of intents based on the intent clustering methods (intent induction). We postulate there are two salient factors for automatic induction of intents: (1) clustering algorithm for intent labeling and (2) user utterance embedding space. We compare existing off-the-shelf clustering models and embeddings based on DSTC11 evaluation. Our extensive experiments demonstrate that the combined selection of utterance embedding and clustering method in the intent induction task should be carefully considered. We also present that pretrained MiniLM with Agglomerative clustering shows significant improvement in NMI, ARI, F1, accuracy and example coverage in intent induction tasks. The source codes are available at https://github.com/Jeiyoon/dstc11-track2.