Calibration-Aware Bayesian Learning
This work addresses the issue of model calibration for users of deep learning systems, offering an incremental improvement by combining existing regularization approaches.
The paper tackles the problem of unreliable uncertainty estimates in deep learning models by proposing calibration-aware Bayesian neural networks (CA-BNNs), which integrate data-dependent and data-independent regularizers to improve calibration, as validated by reductions in expected calibration error (ECE) and reliability diagrams.
Deep learning models, including modern systems like large language models, are well known to offer unreliable estimates of the uncertainty of their decisions. In order to improve the quality of the confidence levels, also known as calibration, of a model, common approaches entail the addition of either data-dependent or data-independent regularization terms to the training loss. Data-dependent regularizers have been recently introduced in the context of conventional frequentist learning to penalize deviations between confidence and accuracy. In contrast, data-independent regularizers are at the core of Bayesian learning, enforcing adherence of the variational distribution in the model parameter space to a prior density. The former approach is unable to quantify epistemic uncertainty, while the latter is severely affected by model misspecification. In light of the limitations of both methods, this paper proposes an integrated framework, referred to as calibration-aware Bayesian neural networks (CA-BNNs), that applies both regularizers while optimizing over a variational distribution as in Bayesian learning. Numerical results validate the advantages of the proposed approach in terms of expected calibration error (ECE) and reliability diagrams.