Going beyond research datasets: Novel intent discovery in the industry setting
This work addresses the gap between research and industry applications for intent discovery in e-commerce, offering incremental improvements tailored to real-life data.
The paper tackled the problem of novel intent discovery in real-world industry settings, where existing methods relying on public datasets with only question fields underperform, and achieved up to a 33 percentage point performance boost over state-of-the-art models by leveraging in-domain pre-training and conversational structure.
Novel intent discovery automates the process of grouping similar messages (questions) to identify previously unknown intents. However, current research focuses on publicly available datasets which have only the question field and significantly differ from real-life datasets. This paper proposes methods to improve the intent discovery pipeline deployed in a large e-commerce platform. We show the benefit of pre-training language models on in-domain data: both self-supervised and with weak supervision. We also devise the best method to utilize the conversational structure (i.e., question and answer) of real-life datasets during fine-tuning for clustering tasks, which we call Conv. All our methods combined to fully utilize real-life datasets give up to 33pp performance boost over state-of-the-art Constrained Deep Adaptive Clustering (CDAC) model for question only. By comparison CDAC model for the question data only gives only up to 13pp performance boost over the naive baseline.