Gradient-based Intra-attention Pruning on Pre-trained Language Models
This addresses the problem of model efficiency for NLP practitioners, offering an incremental improvement in pruning techniques.
The paper tackles the computational expense of pre-trained language models by proposing GRAIN, a structured pruning method that prunes intra-attention structures with knowledge distillation, achieving 6-7x speedups while maintaining 93-99% performance on tasks like GLUE and SQuAD.
Pre-trained language models achieve superior performance but are computationally expensive. Techniques such as pruning and knowledge distillation have been developed to reduce their sizes and latencies. In this work, we propose a structured pruning method GRAIN (Gradient-based Intra-attention pruning), which performs task-specific pruning with knowledge distillation and yields highly effective models. Different from common approaches that prune each attention head as a whole, GRAIN inspects and prunes intra-attention structures, which greatly expands the structure search space and enables more flexible models. We also propose a gradient separation strategy that reduces the interference of distillation on pruning for a better combination of the two approaches. Experiments on GLUE, SQuAD, and CoNLL 2003 show that GRAIN notably outperforms other methods, especially in the high sparsity regime, and achieves $6\sim7\times$ speedups while maintaining $93\%\sim99\%$ performance. Under extreme compression where only $3\%$ transformer weights remain, the pruned model is still competitive compared to larger models.