Joint Text and Label Generation for Spoken Language Understanding
This work addresses generalization in spoken language understanding for applications like virtual assistants, but it is incremental as it builds on existing pretrained models and noisy label techniques.
The authors tackled the problem of limited training data for intent classification and slot labeling by leveraging pretrained language models to generate synthetic text with embedded labels, then training a classifier with mixout regularization to handle noise. Their method achieved superior performance, outperforming the baseline by a large margin.
Generalization is a central problem in machine learning, especially when data is limited. Using prior information to enforce constraints is the principled way of encouraging generalization. In this work, we propose to leverage the prior information embedded in pretrained language models (LM) to improve generalization for intent classification and slot labeling tasks with limited training data. Specifically, we extract prior knowledge from pretrained LM in the form of synthetic data, which encode the prior implicitly. We fine-tune the LM to generate an augmented language, which contains not only text but also encodes both intent labels and slot labels. The generated synthetic data can be used to train a classifier later. Since the generated data may contain noise, we rephrase the learning from generated data as learning with noisy labels. We then utilize the mixout regularization for the classifier and prove its effectiveness to resist label noise in generated data. Empirically, our method demonstrates superior performance and outperforms the baseline by a large margin.