Diverse Embedding Neural Network Language Models
This work addresses language modeling efficiency for natural language processing applications, but it appears incremental as it builds on conventional feed-forward neural network LMs with a modified architecture.
The authors tackled the problem of improving language model performance by proposing a Diverse Embedding Neural Network (DENN) architecture that projects input word history vectors onto multiple diverse low-dimensional sub-spaces, and they demonstrated performance benefits on the Penn Treebank dataset.
We propose Diverse Embedding Neural Network (DENN), a novel architecture for language models (LMs). A DENNLM projects the input word history vector onto multiple diverse low-dimensional sub-spaces instead of a single higher-dimensional sub-space as in conventional feed-forward neural network LMs. We encourage these sub-spaces to be diverse during network training through an augmented loss function. Our language modeling experiments on the Penn Treebank data set show the performance benefit of using a DENNLM.