Simplified TinyBERT: Knowledge Distillation for Document Retrieval
This work addresses efficiency issues in document retrieval for applications requiring fast processing, though it is incremental as it builds on existing knowledge distillation methods.
The paper tackled the high computational cost of BERT models for document ranking by proposing Simplified TinyBERT, which achieved a 15× speedup while significantly outperforming BERT-Base on benchmarks.
Despite the effectiveness of utilizing the BERT model for document ranking, the high computational cost of such approaches limits their uses. To this end, this paper first empirically investigates the effectiveness of two knowledge distillation models on the document ranking task. In addition, on top of the recently proposed TinyBERT model, two simplifications are proposed. Evaluations on two different and widely-used benchmarks demonstrate that Simplified TinyBERT with the proposed simplifications not only boosts TinyBERT, but also significantly outperforms BERT-Base when providing 15$\times$ speedup.