Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents
This work addresses the challenge of data scarcity in SLU for voice-powered agents, offering an incremental improvement over existing methods.
The paper tackles the problem of improving Spoken Language Understanding (SLU) for intelligent agents by leveraging unlabeled utterances through unsupervised transfer learning, resulting in significant performance gains, such as matching the performance of training from scratch with 10-15x more labeled data when using only 1000 labeled samples.
User interaction with voice-powered agents generates large amounts of unlabeled utterances. In this paper, we explore techniques to efficiently transfer the knowledge from these unlabeled utterances to improve model performance on Spoken Language Understanding (SLU) tasks. We use Embeddings from Language Model (ELMo) to take advantage of unlabeled data by learning contextualized word representations. Additionally, we propose ELMo-Light (ELMoL), a faster and simpler unsupervised pre-training method for SLU. Our findings suggest unsupervised pre-training on a large corpora of unlabeled utterances leads to significantly better SLU performance compared to training from scratch and it can even outperform conventional supervised transfer. Additionally, we show that the gains from unsupervised transfer techniques can be further improved by supervised transfer. The improvements are more pronounced in low resource settings and when using only 1000 labeled in-domain samples, our techniques match the performance of training from scratch on 10-15x more labeled in-domain data.