Speeding Up Entmax
This work addresses a computational bottleneck for researchers and practitioners using sparse attention mechanisms in neural networks, though it is incremental as it builds on existing α-entmax.
The paper tackles the speed issue of α-entmax, a sparse alternative to softmax for language processing, by proposing a faster alternative that matches or exceeds machine translation performance while maintaining speed comparable to optimized softmax.
Softmax is the de facto standard in modern neural networks for language processing when it comes to normalizing logits. However, by producing a dense probability distribution each token in the vocabulary has a nonzero chance of being selected at each generation step, leading to a variety of reported problems in text generation. $α$-entmax of Peters et al. (2019, arXiv:1905.05702) solves this problem, but is considerably slower than softmax. In this paper, we propose an alternative to $α$-entmax, which keeps its virtuous characteristics, but is as fast as optimized softmax and achieves on par or better performance in machine translation task.