Dropout Distillation for Efficiently Estimating Model Confidence
This work addresses the need for better-calibrated uncertainty scores in neural networks for tasks like image classification and object detection, offering a more efficient alternative to Bayesian methods, though it is incremental in nature.
The paper tackles the problem of efficiently estimating model confidence in neural networks by proposing Dropout Distillation (DDN), which improves calibration and increases classification accuracy on CIFAR-10, achieving competitive results with 100 Monte Carlo samples while also enhancing calibration in object detection on COCO.
We propose an efficient way to output better calibrated uncertainty scores from neural networks. The Distilled Dropout Network (DDN) makes standard (non-Bayesian) neural networks more introspective by adding a new training loss which prevents them from being overconfident. Our method is more efficient than Bayesian neural networks or model ensembles which, despite providing more reliable uncertainty scores, are more cumbersome to train and slower to test. We evaluate DDN on the the task of image classification on the CIFAR-10 dataset and show that our calibration results are competitive even when compared to 100 Monte Carlo samples from a dropout network while they also increase the classification accuracy. We also propose better calibration within the state of the art Faster R-CNN object detection framework and show, using the COCO dataset, that DDN helps train better calibrated object detectors.