Linear Attention Mechanism: An Efficient Attention for Semantic Segmentation
This work addresses efficiency bottlenecks in attention mechanisms for semantic segmentation, but it appears incremental as it builds on existing attention methods.
The authors tackled the high memory and computational costs of dot-product attention by proposing a Linear Attention Mechanism, achieving efficient performance in semantic segmentation experiments.
In this paper, to remedy this deficiency, we propose a Linear Attention Mechanism which is approximate to dot-product attention with much less memory and computational costs. The efficient design makes the incorporation between attention mechanisms and neural networks more flexible and versatile. Experiments conducted on semantic segmentation demonstrated the effectiveness of linear attention mechanism. Code is available at https://github.com/lironui/Linear-Attention-Mechanism.