r-softmax: Generalized Softmax with Controllable Sparsity Rate
This addresses the problem of non-sparse outputs in probability mapping functions for machine learning practitioners, offering a controllable mechanism for sparsity, though it is incremental as it modifies an existing function.
The paper tackles the limitation of softmax in neural networks by proposing r-softmax, a modification that outputs sparse probability distributions with controllable sparsity rates, showing it outperforms other sparse alternatives on multi-label datasets and improves performance when applied to transformer language models in NLP tasks.
Nowadays artificial neural network models achieve remarkable results in many disciplines. Functions mapping the representation provided by the model to the probability distribution are the inseparable aspect of deep learning solutions. Although softmax is a commonly accepted probability mapping function in the machine learning community, it cannot return sparse outputs and always spreads the positive probability to all positions. In this paper, we propose r-softmax, a modification of the softmax, outputting sparse probability distribution with controllable sparsity rate. In contrast to the existing sparse probability mapping functions, we provide an intuitive mechanism for controlling the output sparsity level. We show on several multi-label datasets that r-softmax outperforms other sparse alternatives to softmax and is highly competitive with the original softmax. We also apply r-softmax to the self-attention module of a pre-trained transformer language model and demonstrate that it leads to improved performance when fine-tuning the model on different natural language processing tasks.