Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models
This work addresses the problem of efficient model deployment for NLP practitioners by providing empirical guidance on student model selection and distillation techniques, though it is incremental as it builds on existing knowledge distillation methods.
The study benchmarked knowledge distillation from task-specific BERT-base teachers to various student models (BiLSTM, CNN, BERT variants) on 12 Indonesian datasets for text classification and sequence labeling, finding that BiLSTM and CNN offer the best performance-resource trade-off compared to pruned BERT models.
We perform knowledge distillation (KD) benchmark from task-specific BERT-base teacher models to various student models: BiLSTM, CNN, BERT-Tiny, BERT-Mini, and BERT-Small. Our experiment involves 12 datasets grouped in two tasks: text classification and sequence labeling in the Indonesian language. We also compare various aspects of distillations including the usage of word embeddings and unlabeled data augmentation. Our experiments show that, despite the rising popularity of Transformer-based models, using BiLSTM and CNN student models provide the best trade-off between performance and computational resource (CPU, RAM, and storage) compared to pruned BERT models. We further propose some quick wins on performing KD to produce small NLP models via efficient KD training mechanisms involving simple choices of loss functions, word embeddings, and unlabeled data preparation.