Active Learning for New Domains in Natural Language Understanding
This work addresses the challenge of efficiently adapting NLU systems to new domains, which is incremental as it builds on existing active learning methods.
The paper tackled the problem of improving natural language understanding (NLU) accuracy for new domains using active learning, achieving 6.6%-9% relative error rate reduction compared to random sampling and 4.6%-9% improvement in human-in-the-loop case studies.
We explore active learning (AL) for improving the accuracy of new domains in a natural language understanding (NLU) system. We propose an algorithm called Majority-CRF that uses an ensemble of classification models to guide the selection of relevant utterances, as well as a sequence labeling model to help prioritize informative examples. Experiments with three domains show that Majority-CRF achieves 6.6%-9% relative error rate reduction compared to random sampling with the same annotation budget, and statistically significant improvements compared to other AL approaches. Additionally, case studies with human-in-the-loop AL on six new domains show 4.6%-9% improvement on an existing NLU system.