$ε$-Softmax: Approximating One-Hot Vectors for Mitigating Label Noise
This work addresses label noise mitigation for deep learning practitioners, offering an incremental improvement by combining ε-softmax with symmetric losses for better robustness and learning trade-offs.
The paper tackles the problem of noisy labels in deep neural networks by proposing ε-softmax, which approximates one-hot vectors with a controllable error to relax the symmetric condition in robust loss functions, achieving noise-tolerant learning with theoretical guarantees and demonstrating superiority in experiments on synthetic and real-world noise.
Noisy labels pose a common challenge for training accurate deep neural networks. To mitigate label noise, prior studies have proposed various robust loss functions to achieve noise tolerance in the presence of label noise, particularly symmetric losses. However, they usually suffer from the underfitting issue due to the overly strict symmetric condition. In this work, we propose a simple yet effective approach for relaxing the symmetric condition, namely $ε$-softmax, which simply modifies the outputs of the softmax layer to approximate one-hot vectors with a controllable error $ε$. Essentially, $ε$-softmax not only acts as an alternative for the softmax layer, but also implicitly plays the crucial role in modifying the loss function. We prove theoretically that $ε$-softmax can achieve noise-tolerant learning with controllable excess risk bound for almost any loss function. Recognizing that $ε$-softmax-enhanced losses may slightly reduce fitting ability on clean datasets, we further incorporate them with one symmetric loss, thereby achieving a better trade-off between robustness and effective learning. Extensive experiments demonstrate the superiority of our method in mitigating synthetic and real-world label noise. The code is available at https://github.com/cswjl/eps-softmax.