MSWA: Refining Local Attention with Multi-ScaleWindow Attention
This addresses a bottleneck in local attention mechanisms for large language models, offering an incremental improvement in efficiency and effectiveness for NLP applications.
The paper tackles the inefficiency of uniform window sizes in sliding window attention for Transformers by proposing Multi-Scale Window Attention (MSWA), which uses diverse window sizes across heads and layers, resulting in improved performance on language modeling and common-sense reasoning tasks.
Transformer-based LLMs have achieved exceptional performance across a wide range of NLP tasks. However, the standard self-attention mechanism suffers from quadratic time complexity and linearly increased cache size. Sliding window attention (SWA) solves this problem by restricting the attention range to a fixed-size local context window. Nevertheless, SWA employs a uniform window size for each head in each layer, making it inefficient in capturing context of varying scales. To mitigate this limitation, we propose Multi-Scale Window Attention (MSWA) which applies diverse window sizes across heads and layers in the Transformer. It not only allows for different window sizes among heads within the same layer but also progressively increases window size allocation from shallow to deep layers, thus enabling the model to capture contextual information with different lengths and distances. Experimental results on language modeling and common-sense reasoning tasks substantiate that MSWA outperforms traditional local attention in both effectiveness and efficiency.