Replacing softmax with ReLU in Vision Transformers
This addresses a method improvement for vision transformer efficiency, but it is incremental as it builds on known issues with activation functions.
The paper tackled the problem of accuracy degradation when replacing softmax with ReLU in vision transformers by introducing division by sequence length, finding that ReLU-attention can approach or match softmax-attention performance in scaling behavior on ImageNet-21k.
Previous research observed accuracy degradation when replacing the attention softmax with a point-wise activation such as ReLU. In the context of vision transformers, we find that this degradation is mitigated when dividing by sequence length. Our experiments training small to large vision transformers on ImageNet-21k indicate that ReLU-attention can approach or match the performance of softmax-attention in terms of scaling behavior as a function of compute.