Circling Back to Recurrent Models of Language
This work addresses the challenge of making recurrent models more effective for language modeling, though it is incremental in nature.
The paper tackles the problem of improving purely recurrent models for language modeling, achieving a new state of the art on small datasets and Enwik8 with dynamic evaluation.
Just because some purely recurrent models suffer from being hard to optimize and inefficient on today's hardware, they are not necessarily bad models of language. We demonstrate this by the extent to which these models can still be improved by a combination of a slightly better recurrent cell, architecture, objective, as well as optimization. In the process, we establish a new state of the art for language modelling on small datasets and on Enwik8 with dynamic evaluation.