Uncertainty Characteristics Curves: A Systematic Assessment of Prediction Intervals
This addresses the need for comprehensive uncertainty quantification in AI, particularly for regression tasks, offering a tool to improve trust and comparability across studies.
The paper tackles the difficulty in evaluating and comparing prediction intervals for model uncertainty in regression by introducing a novel assessment methodology that is operating point agnostic, leveraging operating characteristics curves and gain over a reference to provide a systematic approach.
Accurate quantification of model uncertainty has long been recognized as a fundamental requirement for trusted AI. In regression tasks, uncertainty is typically quantified using prediction intervals calibrated to a specific operating point, making evaluation and comparison across different studies difficult. Our work leverages: (1) the concept of operating characteristics curves and (2) the notion of a gain over a simple reference, to derive a novel operating point agnostic assessment methodology for prediction intervals. The paper describes the corresponding algorithm, provides a theoretical analysis, and demonstrates its utility in multiple scenarios. We argue that the proposed method addresses the current need for comprehensive assessment of prediction intervals and thus represents a valuable addition to the uncertainty quantification toolbox.