EcoTransformer: Attention without Multiplication
This addresses energy efficiency for AI practitioners using Transformers, but it is incremental as it modifies an existing bottleneck without a paradigm shift.
The authors tackled the high computational and energy costs of Transformer's scaled dot-product attention by proposing EcoTransformer, which uses a Laplacian kernel with L1 distances to eliminate matrix multiplication. It achieved performance on par or better in NLP, bioinformatics, and vision tasks while significantly reducing energy consumption.
The Transformer, with its scaled dot-product attention mechanism, has become a foundational architecture in modern AI. However, this mechanism is computationally intensive and incurs substantial energy costs. We propose a new Transformer architecture EcoTransformer, in which the output context vector is constructed as the convolution of the values using a Laplacian kernel, where the distances are measured by the L1 metric between the queries and keys. Compared to dot-product based attention, the new attention score calculation is free of matrix multiplication. It performs on par with, or even surpasses, scaled dot-product attention in NLP, bioinformatics, and vision tasks, while consuming significantly less energy. (This version (v2) supersedes v1 and reflects the intended release and licensing.)