SDBERT: SparseDistilBERT, a faster and smaller BERT model
This work addresses efficiency issues for users of large language models like BERT, though it is incremental as it builds on existing techniques.
The authors tackled the problem of reducing BERT's computational and memory costs by introducing SparseDistilBERT, which combines sparse attention and knowledge distillation to achieve a 60% size reduction, retain 97% performance, and reduce training time by 40%.
In this work we introduce a new transformer architecture called SparseDistilBERT (SDBERT), which is a combination of sparse attention and knowledge distillantion (KD). We implemented sparse attention mechanism to reduce quadratic dependency on input length to linear. In addition to reducing computational complexity of the model, we used knowledge distillation (KD). We were able to reduce the size of BERT model by 60% while retaining 97% performance and it only took 40% of time to train.