Effectiveness of Hierarchical Softmax in Large Scale Classification Tasks
This addresses efficiency issues for practitioners handling large-scale classification tasks, but it is incremental as it evaluates an existing method on new data.
The paper tackled the computational expense of Softmax in large-scale classification by evaluating Hierarchical Softmax on LSHTC datasets, finding that its performance degrades as the number of classes increases, with macro F1 scores showing this decline.
Typically, Softmax is used in the final layer of a neural network to get a probability distribution for output classes. But the main problem with Softmax is that it is computationally expensive for large scale data sets with large number of possible outputs. To approximate class probability efficiently on such large scale data sets we can use Hierarchical Softmax. LSHTC datasets were used to study the performance of the Hierarchical Softmax. LSHTC datasets have large number of categories. In this paper we evaluate and report the performance of normal Softmax Vs Hierarchical Softmax on LSHTC datasets. This evaluation used macro f1 score as a performance measure. The observation was that the performance of Hierarchical Softmax degrades as the number of classes increase.