schuBERT: Optimizing Elements of BERT
This work addresses the computational prohibitive nature of BERT for natural language processing applications, offering a more efficient alternative with improved performance, though it is incremental as it builds on existing BERT architecture.
The paper tackled the problem of BERT's computational and parameter inefficiency by revisiting its architecture choices to create a lighter model, resulting in schuBERT achieving 6.6% higher average accuracy on GLUE and SQuAD datasets compared to BERT with three encoder layers while maintaining the same number of parameters.
Transformers \citep{vaswani2017attention} have gradually become a key component for many state-of-the-art natural language representation models. A recent Transformer based model- BERT \citep{devlin2018bert} achieved state-of-the-art results on various natural language processing tasks, including GLUE, SQuAD v1.1, and SQuAD v2.0. This model however is computationally prohibitive and has a huge number of parameters. In this work we revisit the architecture choices of BERT in efforts to obtain a lighter model. We focus on reducing the number of parameters yet our methods can be applied towards other objectives such FLOPs or latency. We show that much efficient light BERT models can be obtained by reducing algorithmically chosen correct architecture design dimensions rather than reducing the number of Transformer encoder layers. In particular, our schuBERT gives $6.6\%$ higher average accuracy on GLUE and SQuAD datasets as compared to BERT with three encoder layers while having the same number of parameters.