Variational Learning Induces Adaptive Label Smoothing
This work provides a new way to handle overconfident predictions in machine learning, though it is incremental as it connects existing Bayesian methods to label smoothing.
The paper tackles the problem of overconfident predictions by showing that variational learning induces adaptive label smoothing, which handles labeling errors and distribution shifts. The variational algorithm IVON outperforms traditional label smoothing, with empirical results demonstrating its effectiveness.
We show that variational learning naturally induces an adaptive label smoothing where label noise is specialized for each example. Such label-smoothing is useful to handle examples with labeling errors and distribution shifts, but designing a good adaptivity strategy is not always easy. We propose to skip this step and simply use the natural adaptivity induced during the optimization of a variational objective. We show empirical results where a variational algorithm called IVON outperforms traditional label smoothing and yields adaptivity strategies similar to those of an existing approach. By connecting Bayesian methods to label smoothing, our work provides a new way to handle overconfident predictions.