Midpoint Regularization: from High Uncertainty Training to Conservative Classification
This incremental improvement addresses generalization issues in classification models for machine learning practitioners.
The paper tackles the problem of overconfident predictions in classification by extending Label Smoothing to use midpoint samples from random pairs, achieving up to 30% relative reduction in classification error.
Label Smoothing (LS) improves model generalization through penalizing models from generating overconfident output distributions. For each training sample the LS strategy smooths the one-hot encoded training signal by distributing its distribution mass over the non-ground truth classes. We extend this technique by considering example pairs, coined PLS. PLS first creates midpoint samples by averaging random sample pairs and then learns a smoothing distribution during training for each of these midpoint samples, resulting in midpoints with high uncertainty labels for training. We empirically show that PLS significantly outperforms LS, achieving up to 30% of relative classification error reduction. We also visualize that PLS produces very low winning softmax scores for both in and out of distribution samples.