Masked ELMo: An evolution of ELMo towards fully contextual RNN language models
This work addresses the need for more efficient and effective contextual RNN language models for natural language processing tasks, representing an incremental improvement over ELMo.
The paper tackles the problem of limited context in RNN-based language models by introducing Masked ELMo, which learns fully bidirectional word representations using a masked language model objective and optimizations for faster training. The result is a model that significantly outperforms ELMo on the GLUE benchmark while maintaining low computational cost, achieving competitive performance with transformer approaches.
This paper presents Masked ELMo, a new RNN-based model for language model pre-training, evolved from the ELMo language model. Contrary to ELMo which only uses independent left-to-right and right-to-left contexts, Masked ELMo learns fully bidirectional word representations. To achieve this, we use the same Masked language model objective as BERT. Additionally, thanks to optimizations on the LSTM neuron, the integration of mask accumulation and bidirectional truncated backpropagation through time, we have increased the training speed of the model substantially. All these improvements make it possible to pre-train a better language model than ELMo while maintaining a low computational cost. We evaluate Masked ELMo by comparing it to ELMo within the same protocol on the GLUE benchmark, where our model outperforms significantly ELMo and is competitive with transformer approaches.