Dynamic Evaluation of Neural Sequence Models
This work addresses the challenge of enhancing sequence modeling for natural language processing tasks, offering incremental improvements over existing adaptation methods.
The paper tackles the problem of improving neural sequence models by adapting them to recent history using dynamic evaluation, resulting in state-of-the-art performance with word-level perplexities of 51.1 on Penn Treebank and 44.3 on WikiText-2, and character-level cross-entropies of 1.19 bits/char on text8 and 1.08 bits/char on Hutter Prize.
We present methodology for using dynamic evaluation to improve neural sequence models. Models are adapted to recent history via a gradient descent based mechanism, causing them to assign higher probabilities to re-occurring sequential patterns. Dynamic evaluation outperforms existing adaptation approaches in our comparisons. Dynamic evaluation improves the state-of-the-art word-level perplexities on the Penn Treebank and WikiText-2 datasets to 51.1 and 44.3 respectively, and the state-of-the-art character-level cross-entropies on the text8 and Hutter Prize datasets to 1.19 bits/char and 1.08 bits/char respectively.