Improving BERT with Hybrid Pooling Network and Drop Mask
This work addresses efficiency and performance issues in BERT for natural language processing, but it is incremental as it builds on existing transformer models.
The authors tackled the problem of BERT's uniform self-attention across layers and the mismatch from excessive mask tokens in pre-training, resulting in HybridBERT achieving lower loss, 8% faster training, 13% lower memory cost, and 1.5% higher accuracy on downstream tasks, with DropMask further improving accuracies across masking rates.
Transformer-based pre-trained language models, such as BERT, achieve great success in various natural language understanding tasks. Prior research found that BERT captures a rich hierarchy of linguistic information at different layers. However, the vanilla BERT uses the same self-attention mechanism for each layer to model the different contextual features. In this paper, we propose a HybridBERT model which combines self-attention and pooling networks to encode different contextual features in each layer. Additionally, we propose a simple DropMask method to address the mismatch between pre-training and fine-tuning caused by excessive use of special mask tokens during Masked Language Modeling pre-training. Experiments show that HybridBERT outperforms BERT in pre-training with lower loss, faster training speed (8% relative), lower memory cost (13% relative), and also in transfer learning with 1.5% relative higher accuracies on downstream tasks. Additionally, DropMask improves accuracies of BERT on downstream tasks across various masking rates.