Distance-Based Learning from Errors for Confidence Calibration
This addresses the issue of unreliable confidence estimates in DNNs for applications requiring trustworthy predictions, representing an incremental improvement over existing calibration techniques.
The paper tackled the problem of poor confidence calibration in deep neural networks by proposing a distance-based learning from errors (DBLE) method, which outperformed alternative single-model approaches and achieved comparable performance to ensemble methods with lower computational cost and parameters.
Deep neural networks (DNNs) are poorly calibrated when trained in conventional ways. To improve confidence calibration of DNNs, we propose a novel training method, distance-based learning from errors (DBLE). DBLE bases its confidence estimation on distances in the representation space. In DBLE, we first adapt prototypical learning to train classification models. It yields a representation space where the distance between a test sample and its ground truth class center can calibrate the model's classification performance. At inference, however, these distances are not available due to the lack of ground truth labels. To circumvent this by inferring the distance for every test sample, we propose to train a confidence model jointly with the classification model. We integrate this into training by merely learning from mis-classified training samples, which we show to be highly beneficial for effective learning. On multiple datasets and DNN architectures, we demonstrate that DBLE outperforms alternative single-model confidence calibration approaches. DBLE also achieves comparable performance with computationally-expensive ensemble approaches with lower computational cost and lower number of parameters.