Character n-gram Embeddings to Improve RNN Language Models
This work addresses language modeling for NLP applications, offering incremental improvements by integrating character-level information into existing word embedding methods.
The paper tackled the problem of improving RNN language models by incorporating character n-gram embeddings, achieving state-of-the-art perplexities on Penn Treebank, WikiText-2, and WikiText-103 datasets, and showing positive effects on machine translation and headline generation tasks.
This paper proposes a novel Recurrent Neural Network (RNN) language model that takes advantage of character information. We focus on character n-grams based on research in the field of word embedding construction (Wieting et al. 2016). Our proposed method constructs word embeddings from character n-gram embeddings and combines them with ordinary word embeddings. We demonstrate that the proposed method achieves the best perplexities on the language modeling datasets: Penn Treebank, WikiText-2, and WikiText-103. Moreover, we conduct experiments on application tasks: machine translation and headline generation. The experimental results indicate that our proposed method also positively affects these tasks.