TrimBERT: Tailoring BERT for Trade-offs
This work addresses the computational resource limitations for NLP practitioners, but it is incremental as it builds on existing BERT architectures with specific optimizations.
The paper tackles the problem of BERT models being computationally expensive by reducing intermediate layers and simplifying operations, resulting in minimal accuracy loss (e.g., fine-tuning accuracy loss of less than 1% on some tasks) while significantly decreasing model size and training time by up to 30%.
Models based on BERT have been extremely successful in solving a variety of natural language processing (NLP) tasks. Unfortunately, many of these large models require a great deal of computational resources and/or time for pre-training and fine-tuning which limits wider adoptability. While self-attention layers have been well-studied, a strong justification for inclusion of the intermediate layers which follow them remains missing in the literature. In this work, we show that reducing the number of intermediate layers in BERT-Base results in minimal fine-tuning accuracy loss of downstream tasks while significantly decreasing model size and training time. We further mitigate two key bottlenecks, by replacing all softmax operations in the self-attention layers with a computationally simpler alternative and removing half of all layernorm operations. This further decreases the training time while maintaining a high level of fine-tuning accuracy.