Crisp Attention: Regularizing Transformers via Structured Sparsity
This work addresses computational efficiency and overfitting in Transformer models, showing that sparsity can enhance performance rather than degrade it, though it is incremental as it builds on existing sparsity techniques.
The paper tackles the quadratic computational cost of self-attention in Transformers by introducing structured sparsity during fine-tuning, finding that 80% attention sparsity improves validation accuracy by 0.97% on the SST-2 sentiment analysis task.
The quadratic computational cost of the self-attention mechanism is a primary challenge in scaling Transformer models. While attention sparsity is widely studied as a technique to improve computational efficiency, it is almost universally assumed to come at the cost of model accuracy. In this paper, we report a surprising counter-example to this common wisdom. By introducing structured, post-hoc sparsity to the attention mechanism of a DistilBERT model during fine-tuning on the SST-2 sentiment analysis task, we find that model accuracy improves significantly. Our model with 80\% attention sparsity achieves a validation accuracy of 91.59\%, a 0.97\% absolute improvement over the dense baseline. We hypothesize that this phenomenon is due to sparsity acting as a powerful implicit regularizer, preventing the model from overfitting by forcing it to make predictions with a more constrained and robust set of features. Our work recasts attention sparsity not just as a tool for computational efficiency, but as a potential method for improving the generalization and performance of Transformer models.