Escaping the Gradient Vanishing: Periodic Alternatives of Softmax in Attention Mechanism
This work addresses a critical bottleneck in transformers for machine learning practitioners, offering a novel solution to gradient issues in deep structures.
The paper tackles the gradient vanishing problem in attention mechanisms by replacing the exponential function in Softmax with periodic functions, achieving substantial improvements in gradient stability and performance compared to Softmax and its variants in experiments on a LeViT-inspired demo.
Softmax is widely used in neural networks for multiclass classification, gate structure and attention mechanisms. The statistical assumption that the input is normal distributed supports the gradient stability of Softmax. However, when used in attention mechanisms such as transformers, since the correlation scores between embeddings are often not normally distributed, the gradient vanishing problem appears, and we prove this point through experimental confirmation. In this work, we suggest that replacing the exponential function by periodic functions, and we delve into some potential periodic alternatives of Softmax from the view of value and gradient. Through experiments on a simply designed demo referenced to LeViT, our method is proved to be able to alleviate the gradient problem and yield substantial improvements compared to Softmax and its variants. Further, we analyze the impact of pre-normalization for Softmax and our methods through mathematics and experiments. Lastly, we increase the depth of the demo and prove the applicability of our method in deep structures.