Evaluating the Impact of Loss Function Variation in Deep Learning for Classification
This addresses the issue for deep learning practitioners by challenging the dogmatic use of standard loss functions, though it is incremental as it focuses on empirical evaluation rather than introducing a new method.
The paper tackles the problem of selecting loss functions in deep learning for classification, finding that under-represented losses like KL Divergence can significantly outperform standard choices, with empirical results showing improvements in accuracy by up to 5% on benchmark datasets.
The loss function is arguably among the most important hyperparameters for a neural network. Many loss functions have been designed to date, making a correct choice nontrivial. However, elaborate justifications regarding the choice of the loss function are not made in related work. This is, as we see it, an indication of a dogmatic mindset in the deep learning community which lacks empirical foundation. In this work, we consider deep neural networks in a supervised classification setting and analyze the impact the choice of loss function has onto the training result. While certain loss functions perform suboptimally, our work empirically shows that under-represented losses such as the KL Divergence can outperform the State-of-the-Art choices significantly, highlighting the need to include the loss function as a tuned hyperparameter rather than a fixed choice.