Differentiable Window for Dynamic Local Attention
This work addresses the need for more focused attention in NLP models, offering a general-purpose component that enhances standard attention modules, though it appears incremental as it builds upon existing Transformer architectures.
The authors tackled the problem of improving attention mechanisms in Transformers by introducing Differentiable Window, a module for dynamic window selection, which led to consistent and sizable performance gains across multiple NLP tasks.
We propose Differentiable Window, a new neural module and general purpose component for dynamic window selection. While universally applicable, we demonstrate a compelling use case of utilizing Differentiable Window to improve standard attention modules by enabling more focused attentions over the input regions. We propose two variants of Differentiable Window, and integrate them within the Transformer architecture in two novel ways. We evaluate our proposed approach on a myriad of NLP tasks, including machine translation, sentiment analysis, subject-verb agreement and language modeling. Our experimental results demonstrate consistent and sizable improvements across all tasks.