Universal Training of Neural Networks to Achieve Bayes Optimal Classification Accuracy
This addresses the challenge of improving generalization in neural networks for classification tasks, though it appears incremental as it builds on existing divergence concepts.
The paper tackles the problem of achieving Bayes optimal classification accuracy by introducing a novel loss function called BOLT, which enforces models to reach the Bayes error rate, and demonstrates that it performs on par with or exceeds cross-entropy on datasets like MNIST and CIFAR-10.
This work invokes the notion of $f$-divergence to introduce a novel upper bound on the Bayes error rate of a general classification task. We show that the proposed bound can be computed by sampling from the output of a parameterized model. Using this practical interpretation, we introduce the Bayes optimal learning threshold (BOLT) loss whose minimization enforces a classification model to achieve the Bayes error rate. We validate the proposed loss for image and text classification tasks, considering MNIST, Fashion-MNIST, CIFAR-10, and IMDb datasets. Numerical experiments demonstrate that models trained with BOLT achieve performance on par with or exceeding that of cross-entropy, particularly on challenging datasets. This highlights the potential of BOLT in improving generalization.