HINT3: Raising the bar for Intent Detection in the Wild
This work addresses the challenge of imbalanced and correlated data in intent detection for chatbot applications, but it is incremental as it focuses on dataset creation and benchmarking without proposing a new method.
The authors tackled the problem of intent detection in real-world scenarios by introducing three new datasets from live chatbots to benchmark systems more accurately, finding that existing NLU platforms and a BERT-based classifier perform inadequately due to reliance on unintended patterns in training data.
Intent Detection systems in the real world are exposed to complexities of imbalanced datasets containing varying perception of intent, unintended correlations and domain-specific aberrations. To facilitate benchmarking which can reflect near real-world scenarios, we introduce 3 new datasets created from live chatbots in diverse domains. Unlike most existing datasets that are crowdsourced, our datasets contain real user queries received by the chatbots and facilitates penalising unwanted correlations grasped during the training process. We evaluate 4 NLU platforms and a BERT based classifier and find that performance saturates at inadequate levels on test sets because all systems latch on to unintended patterns in training data.