Cost-Effective Attention Mechanisms for Low Resource Settings: Necessity & Sufficiency of Linear Transformations
This work addresses efficiency challenges for deploying attention mechanisms in resource-constrained environments, such as mobile or edge devices, though it is incremental as it builds on existing SDPA frameworks.
The paper tackled the high memory and computational costs of Scaled Dot Product Attention in low-resource settings by proposing three attention variants that remove or add linear transformations, resulting in models with 25-50% fewer parameters and negligible performance loss, with one variant outperforming SDPA by up to 10% while improving speed and reducing parameters by 25%.
From natural language processing to vision, Scaled Dot Product Attention (SDPA) is the backbone of most modern deep learning applications. Unfortunately, its memory and computational requirements can be prohibitive in low-resource settings. In this paper, we improve its efficiency without sacrificing its versatility. We propose three attention variants where we remove consecutive linear transformations or add a novel one, and evaluate them on a range of standard NLP and vision tasks. Our proposed models are substantially lighter than standard SDPA (and have 25-50% fewer parameters). We show that the performance cost of these changes is negligible relative to size reduction and that in one case (Super Attention) we succeed in outperforming SDPA by up to 10% while improving its speed and reducing its parameters by 25%.