SAP: Syntactic Attention Pruning for Transformer-based Language Models
This addresses model compression for transformer-based language models, offering a novel approach that improves interpretability and robustness, though it is incremental in building on existing pruning strategies.
The paper tackles the problem of pruning attention heads in Transformer models by introducing Syntactic Attention Pruning (SAP), which uses syntactic structure and attention patterns to guide pruning, achieving performance comparable to state-of-the-art methods and enhancing interpretability.
This paper introduces Syntactic Attention Pruning (SAP), a novel method for effectively pruning attention heads in Transformer models. Unlike conventional approaches that rely solely on mathematical analysis of model weights and activations, SAP incorporates both the syntactic structure and attention patterns of sentences to guide the pruning process. By leveraging these linguistic features, SAP not only achieves performance comparable to state-of-the-art methods but also enhances the interpretability of model behavior. To further improve robustness, we propose Candidate Filtering (CF), a mechanism that prioritizes heads based on their contribution to model performance, mitigating degradation during pruning. Experimental results indicate that SAP effectively preserves critical heads of a high density of strong attention values, outperforming existing head pruning strategies in retrain-free settings. These findings position SAP as a promising foundation for a new direction in model compression research, offering high flexibility for pruning across all transformer-based language models.