Exploring Alternatives to Softmax Function
This work provides incremental improvements for researchers and practitioners working on multiclass classification and attention mechanisms by offering potentially more effective activation functions.
This paper explores alternatives to the softmax function, including Taylor softmax, soft-margin softmax (SM-softmax), and a proposed SM-Taylor softmax. The authors found that a configuration of the SM-Taylor softmax consistently outperformed the standard softmax and other alternatives in image classification tasks across different datasets.
Softmax function is widely used in artificial neural networks for multiclass classification, multilabel classification, attention mechanisms, etc. However, its efficacy is often questioned in literature. The log-softmax loss has been shown to belong to a more generic class of loss functions, called spherical family, and its member log-Taylor softmax loss is arguably the best alternative in this class. In another approach which tries to enhance the discriminative nature of the softmax function, soft-margin softmax (SM-softmax) has been proposed to be the most suitable alternative. In this work, we investigate Taylor softmax, SM-softmax and our proposed SM-Taylor softmax, an amalgamation of the earlier two functions, as alternatives to softmax function. Furthermore, we explore the effect of expanding Taylor softmax up to ten terms (original work proposed expanding only to two terms) along with the ramifications of considering Taylor softmax to be a finite or infinite series during backpropagation. Our experiments for the image classification task on different datasets reveal that there is always a configuration of the SM-Taylor softmax function that outperforms the normal softmax function and its other alternatives.