Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization
This addresses calibration issues in safety-critical applications like medical diagnosis and autonomous driving, offering incremental improvements over existing methods.
The paper tackles the problem of poor calibration and overconfidence in deep neural networks, showing that sharpness-aware minimization (SAM) reduces calibration error and proposing a variant, CSAM, which achieves even lower error on datasets like ImageNet-1K.
Deep neural networks have been increasingly used in safety-critical applications such as medical diagnosis and autonomous driving. However, many studies suggest that they are prone to being poorly calibrated and have a propensity for overconfidence, which may have disastrous consequences. In this paper, unlike standard training such as stochastic gradient descent, we show that the recently proposed sharpness-aware minimization (SAM) counteracts this tendency towards overconfidence. The theoretical analysis suggests that SAM allows us to learn models that are already well-calibrated by implicitly maximizing the entropy of the predictive distribution. Inspired by this finding, we further propose a variant of SAM, coined as CSAM, to ameliorate model calibration. Extensive experiments on various datasets, including ImageNet-1K, demonstrate the benefits of SAM in reducing calibration error. Meanwhile, CSAM performs even better than SAM and consistently achieves lower calibration error than other approaches