Evaluating and Calibrating Uncertainty Prediction in Regression Tasks
This work addresses the need for reliable uncertainty prediction in safety-critical machine learning applications, though it is incremental as it builds on prior definitions and methods.
The authors tackled the problem of evaluating and calibrating uncertainty predictions in regression tasks, showing that existing definitions have severe limitations and proposing a new definition and simple calibration method that performs as well as more complex ones, with results demonstrated on synthetic and real-world datasets like COCO and KITTI.
Predicting not only the target but also an accurate measure of uncertainty is important for many machine learning applications and in particular safety-critical ones. In this work we study the calibration of uncertainty prediction for regression tasks which often arise in real-world systems. We show that the existing definition for calibration of a regression uncertainty [Kuleshov et al. 2018] has severe limitations in distinguishing informative from non-informative uncertainty predictions. We propose a new definition that escapes this caveat and an evaluation method using a simple histogram-based approach. Our method clusters examples with similar uncertainty prediction and compares the prediction with the empirical uncertainty on these examples. We also propose a simple, scaling-based calibration method that preforms as well as much more complex ones. We show results on both a synthetic, controlled problem and on the object detection bounding-box regression task using the COCO and KITTI datasets.