Grammatically-Guided Sparse Attention for Efficient and Interpretable Transformers
For researchers seeking efficient and interpretable Transformers, this work offers a linguistically-informed sparsity method that preserves accuracy, though it is incremental as it builds on existing sparse attention techniques.
The paper introduces Grammatically-Guided Sparse Attention, which uses POS tags to create attention masks that reduce computational complexity while maintaining accuracy. On SST-2 with a DistilBERT-like model, hard masking achieves 0.8200 accuracy (same as full attention) and soft masking achieves 0.8165.
The quadratic complexity of self-attention in Transformer models remains a significant bottleneck for processing long sequences and deploying large language models efficiently. For this approach, there has been significant research into Sparse Attention, and Deepseek Sparse Attention has combined various methods of creating segments of tokens to reduce the time complexity. This paper introduces a novel approach, Grammatically-Guided Sparse Attention, which constrains attention computations based on the grammatical roles of tokens. By leveraging Parts-of-Speech (POS) tags, attention masks are dynamically generated that enforce linguistically coherent connections between tokens, reducing the computational graph without sacrificing essential linguistic dependencies. Two masking strategies are proposed and evaluated: a hard mask that strictly allows only predefined grammatical interactions, and a soft mask that biases attention towards these interactions. The experiments, conducted on the SST-2 sentiment classification task using a DistilBERT-like architecture, demonstrate that Grammatically-Guided Sparse Attention maintains comparable accuracy to full attention while significantly reducing the theoretical computational overhead. Preliminary results show accuracy values of 0.8200 for hard masking and 0.8165 for soft masking, closely matching the 0.8200 of full attention, providing a path towards more efficient, interpretable, and linguistically-informed Transformer architectures.