Can depth-adaptive BERT perform better on binary classification tasks
This work addresses efficiency in NLP for practitioners by offering a simple method to reduce BERT's computational overhead, though it is incremental as it builds on existing transfer learning approaches.
The study investigated whether smaller sub-networks of BERT can outperform the full model on binary classification tasks, finding that such sub-networks exist and proposing a method to shrink BERT before fine-tuning, which saves time and storage with minimal accuracy loss.
In light of the success of transferring language models into NLP tasks, we ask whether the full BERT model is always the best and does it exist a simple but effective method to find the winning ticket in state-of-the-art deep neural networks without complex calculations. We construct a series of BERT-based models with different size and compare their predictions on 8 binary classification tasks. The results show there truly exist smaller sub-networks performing better than the full model. Then we present a further study and propose a simple method to shrink BERT appropriately before fine-tuning. Some extended experiments indicate that our method could save time and storage overhead extraordinarily with little even no accuracy loss.