Recurrent Neural Networks with Pre-trained Language Model Embedding for Slot Filling Task
This work addresses data efficiency for researchers and practitioners in natural language processing, though it is incremental as it builds on existing RNN and pre-trained model methods.
The paper tackles the slot filling task in spoken language understanding by incorporating pre-trained language model embeddings into RNN-based models, resulting in significantly reduced labeled training data requirements while maintaining performance levels on the ATIS dataset.
In recent years, Recurrent Neural Networks (RNNs) based models have been applied to the Slot Filling problem of Spoken Language Understanding and achieved the state-of-the-art performances. In this paper, we investigate the effect of incorporating pre-trained language models into RNN based Slot Filling models. Our evaluation on the Airline Travel Information System (ATIS) data corpus shows that we can significantly reduce the size of labeled training data and achieve the same level of Slot Filling performance by incorporating extra word embedding and language model embedding layers pre-trained on unlabeled corpora.