Simple Recurrence Improves Masked Language Models
This work addresses the efficiency and performance of language models for NLP practitioners, but it is incremental as it builds on existing Transformer architectures.
The authors tackled the problem of improving Transformer-based masked language models by incorporating a simple recurrent module, achieving an absolute improvement of 2.1 points averaged across 10 tasks and increased stability in fine-tuning.
In this work, we explore whether modeling recurrence into the Transformer architecture can both be beneficial and efficient, by building an extremely simple recurrent module into the Transformer. We compare our model to baselines following the training and evaluation recipe of BERT. Our results confirm that recurrence can indeed improve Transformer models by a consistent margin, without requiring low-level performance optimizations, and while keeping the number of parameters constant. For example, our base model achieves an absolute improvement of 2.1 points averaged across 10 tasks and also demonstrates increased stability in fine-tuning over a range of learning rates.