Adapting by Pruning: A Case Study on BERT
This work addresses the challenge of efficient model adaptation for NLP practitioners, offering a method that reduces model size without significant performance loss, though it is incremental in the context of existing pruning techniques.
The paper tackles the problem of adapting pre-trained models to downstream tasks by proposing a novel pruning paradigm that removes up to 50% of weights in BERT while maintaining similar performance to fine-tuned full models on GLUE tasks.
Adapting pre-trained neural models to downstream tasks has become the standard practice for obtaining high-quality models. In this work, we propose a novel model adaptation paradigm, adapting by pruning, which prunes neural connections in the pre-trained model to optimise the performance on the target task; all remaining connections have their weights intact. We formulate adapting-by-pruning as an optimisation problem with a differentiable loss and propose an efficient algorithm to prune the model. We prove that the algorithm is near-optimal under standard assumptions and apply the algorithm to adapt BERT to some GLUE tasks. Results suggest that our method can prune up to 50% weights in BERT while yielding similar performance compared to the fine-tuned full model. We also compare our method with other state-of-the-art pruning methods and study the topological differences of their obtained sub-networks.