Dynamic-TinyBERT: Boost TinyBERT's Inference Efficiency by Dynamic Sequence Length
This work addresses efficiency and flexibility issues for deploying transformers in production under limited computational budgets, representing an incremental improvement over TinyBERT.
The paper tackles the problem of TinyBERT's performance drop and lack of flexibility for different computational budgets by introducing Dynamic-TinyBERT, which uses sequence-length reduction and hyperparameter optimization to achieve up to 3.3x speedup with less than 1% accuracy loss while being trained only once.
Limited computational budgets often prevent transformers from being used in production and from having their high accuracy utilized. TinyBERT addresses the computational efficiency by self-distilling BERT into a smaller transformer representation having fewer layers and smaller internal embedding. However, TinyBERT's performance drops when we reduce the number of layers by 50%, and drops even more abruptly when we reduce the number of layers by 75% for advanced NLP tasks such as span question answering. Additionally, a separate model must be trained for each inference scenario with its distinct computational budget. In this work we present Dynamic-TinyBERT, a TinyBERT model that utilizes sequence-length reduction and Hyperparameter Optimization for enhanced inference efficiency per any computational budget. Dynamic-TinyBERT is trained only once, performing on-par with BERT and achieving an accuracy-speedup trade-off superior to any other efficient approaches (up to 3.3x with <1% loss-drop). Upon publication, the code to reproduce our work will be open-sourced.