Screening Is Enough
This addresses a core problem in language modeling by improving efficiency and scalability for long-context tasks, though it is an incremental advancement over existing attention mechanisms.
The paper tackled the limitation of softmax attention in language models by introducing Multiscreen, a mechanism that enables absolute query-key relevance by discarding irrelevant keys via a threshold, resulting in comparable validation loss with 40% fewer parameters, stable optimization at larger learning rates, and up to 3.2x faster inference at long contexts.
A core limitation of standard softmax attention is that it does not define a notion of absolute query--key relevance: attention weights are obtained by redistributing a fixed unit mass across all keys according to their relative scores. As a result, relevance is defined only relative to competing keys, and irrelevant keys cannot be explicitly rejected. We introduce Multiscreen, a language-model architecture built around a mechanism we call screening, which enables absolute query--key relevance. Instead of redistributing attention across all keys, screening evaluates each key against an explicit threshold, discarding irrelevant keys and aggregating the remaining keys, thereby removing global competition among keys. Across experiments, Multiscreen achieves comparable validation loss with approximately 40% fewer parameters than a Transformer baseline, enables stable optimization at substantially larger learning rates, maintains strong performance in long-context perplexity, shows little to no degradation in retrieval performance even far beyond the training context length, and reduces inference latency by up to 3.2$\times$ at 100K context length.