TernaryBERT: Distillation-aware Ultra-low Bit BERT
This work addresses the deployment challenge for BERT models in resource-limited environments, representing an incremental improvement in model compression techniques.
The authors tackled the problem of deploying large BERT models on resource-constrained devices by proposing TernaryBERT, which ternarizes weights to reduce model size, achieving a 14.9x reduction while maintaining comparable performance to the full-precision model on benchmarks like GLUE and SQuAD.
Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks.However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices. In this work, we propose TernaryBERT, which ternarizes the weights in a fine-tuned BERT model. Specifically, we use both approximation-based and loss-aware ternarization methods and empirically investigate the ternarization granularity of different parts of BERT. Moreover, to reduce the accuracy degradation caused by the lower capacity of low bits, we leverage the knowledge distillation technique in the training process. Experiments on the GLUE benchmark and SQuAD show that our proposed TernaryBERT outperforms the other BERT quantization methods, and even achieves comparable performance as the full-precision model while being 14.9x smaller.