Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error
This addresses the calibration issue in neural networks, which is critical for real-world applications, though it appears incremental as it builds on existing calibration strategies.
The paper tackles the problem of neural network calibration by introducing a differentiable surrogate for expected calibration error (DECE) and a meta-learning framework to optimize calibration with respect to hyper-parameters, achieving competitive performance with existing approaches.
Calibration of neural networks is a topical problem that is becoming more and more important as neural networks increasingly underpin real-world applications. The problem is especially noticeable when using modern neural networks, for which there is a significant difference between the confidence of the model and the probability of correct prediction. Various strategies have been proposed to improve calibration, yet accurate calibration remains challenging. We propose a novel framework with two contributions: introducing a new differentiable surrogate for expected calibration error (DECE) that allows calibration quality to be directly optimised, and a meta-learning framework that uses DECE to optimise for validation set calibration with respect to model hyper-parameters. The results show that we achieve competitive performance with existing calibration approaches. Our framework opens up a new avenue and toolset for tackling calibration, which we believe will inspire further work on this important challenge.