Cloze-driven Pretraining of Self-attention Networks
This work addresses language understanding for NLP applications, but it is incremental as it builds on existing transformer and cloze-task approaches, aligning with concurrent models like BERT.
The authors tackled the problem of language understanding by pretraining a bi-directional transformer model using a cloze-style word reconstruction task, achieving significant performance gains including new state-of-the-art results on NER and constituency parsing benchmarks, with improvements consistent with BERT.
We present a new approach for pretraining a bi-directional transformer model that provides significant performance gains across a variety of language understanding problems. Our model solves a cloze-style word reconstruction task, where each word is ablated and must be predicted given the rest of the text. Experiments demonstrate large performance gains on GLUE and new state of the art results on NER as well as constituency parsing benchmarks, consistent with the concurrently introduced BERT model. We also present a detailed analysis of a number of factors that contribute to effective pretraining, including data domain and size, model capacity, and variations on the cloze objective.