Input-to-Output Gate to Improve RNN Language Models
This is an incremental improvement for natural language processing researchers and practitioners working with RNN language models.
The paper tackles the problem of improving Recurrent Neural Network (RNN) language models by proposing a reinforcing method called Input-to-Output Gate (IOG) to refine their output layers, resulting in consistent performance boosts on Penn Treebank and WikiText-2 datasets.
This paper proposes a reinforcing method that refines the output layers of existing Recurrent Neural Network (RNN) language models. We refer to our proposed method as Input-to-Output Gate (IOG). IOG has an extremely simple structure, and thus, can be easily combined with any RNN language models. Our experiments on the Penn Treebank and WikiText-2 datasets demonstrate that IOG consistently boosts the performance of several different types of current topline RNN language models.