Deploying a BERT-based Query-Title Relevance Classifier in a Production System: a View from the Trenches
This addresses the problem of deploying large models in production systems for NLP applications, though it is incremental as it builds on existing BERT and knowledge distillation methods.
The authors tackled the challenge of scaling BERT for low-latency industrial use by optimizing a Query-Title Relevance classifier with a compact model called BertBiLSTM, achieving inference in at most 0.2ms on CPU and exceeding BERT's accuracy and efficiency for this task.
The Bidirectional Encoder Representations from Transformers (BERT) model has been radically improving the performance of many Natural Language Processing (NLP) tasks such as Text Classification and Named Entity Recognition (NER) applications. However, it is challenging to scale BERT for low-latency and high-throughput industrial use cases due to its enormous size. We successfully optimize a Query-Title Relevance (QTR) classifier for deployment via a compact model, which we name BERT Bidirectional Long Short-Term Memory (BertBiLSTM). The model is capable of inferring an input in at most 0.2ms on CPU. BertBiLSTM exceeds the off-the-shelf BERT model's performance in terms of accuracy and efficiency for the aforementioned real-world production task. We achieve this result in two phases. First, we create a pre-trained model, called eBERT, which is the original BERT architecture trained with our unique item title corpus. We then fine-tune eBERT for the QTR task. Second, we train the BertBiLSTM model to mimic the eBERT model's performance through a process called Knowledge Distillation (KD) and show the effect of data augmentation to achieve the resembling goal. Experimental results show that the proposed model outperforms other compact and production-ready models.