Dynamic Evaluation of Transformer Language Models
This incremental improvement enhances language modeling for natural language processing tasks by boosting predictive accuracy on benchmark datasets.
The paper tackled improving language modeling performance by combining Transformers with dynamic evaluation, resulting in state-of-the-art reductions in bits/char and perplexity on datasets like enwik8 (0.99 to 0.94 bits/char) and WikiText-103 (18.3 to 16.4 perplexity).
This research note combines two methods that have recently improved the state of the art in language modeling: Transformers and dynamic evaluation. Transformers use stacked layers of self-attention that allow them to capture long range dependencies in sequential data. Dynamic evaluation fits models to the recent sequence history, allowing them to assign higher probabilities to re-occurring sequential patterns. By applying dynamic evaluation to Transformer-XL models, we improve the state of the art on enwik8 from 0.99 to 0.94 bits/char, text8 from 1.08 to 1.04 bits/char, and WikiText-103 from 18.3 to 16.4 perplexity points.