Beyond One-Hot Labels: Semantic Mixing for Model Calibration
This work addresses the challenge of model calibration for machine learning practitioners by providing a novel method to create synthetic uncertain datasets, though it is incremental as it builds on existing calibration and diffusion model techniques.
The paper tackles the problem of model calibration by addressing the lack of datasets with ground-truth uncertainty values, introducing a calibration-aware data augmentation framework that generates synthetic samples with mixed class characteristics and confidence scores using diffusion models, resulting in superior calibration performance compared to state-of-the-art approaches.
Model calibration seeks to ensure that models produce confidence scores that accurately reflect the true likelihood of their predictions being correct. However, existing calibration approaches are fundamentally tied to datasets of one-hot labels implicitly assuming full certainty in all the annotations. Such datasets are effective for classification but provides insufficient knowledge of uncertainty for model calibration, necessitating the curation of datasets with numerically rich ground-truth confidence values. However, due to the scarcity of uncertain visual examples, such samples are not easily available as real datasets. In this paper, we introduce calibration-aware data augmentation to create synthetic datasets of diverse samples and their ground-truth uncertainty. Specifically, we present \textbf{Calibration-aware Semantic Mixing (CSM)}, a novel framework that generates training samples with mixed class characteristics and annotates them with distinct confidence scores via diffusion models. Based on this framework, we propose calibrated reannotation to tackle the misalignment between the annotated confidence score and the mixing ratio during the diffusion reverse process. Besides, we explore the loss functions that better fit the new data representation paradigm. Experimental results demonstrate that CSM achieves superior calibration compared to the state-of-the-art calibration approaches. Our code is \href{https://github.com/E-Galois/CSM}{available here}.