Structure and Distribution Metric for Quantifying the Quality of Uncertainty: Assessing Gaussian Processes, Deep Neural Nets, and Deep Neural Operators for Regression
This work addresses the need for better uncertainty assessment in machine learning models, particularly for regression tasks, but it is incremental as it builds on existing methods with new metrics.
The authors tackled the problem of quantifying uncertainty quality in regression by proposing two bounded metrics for structure and distribution, and applied them to Gaussian Processes, Deep Neural Nets, and Deep Neural Operators, finding that DNNs and DNOs show encouraging results in high-dimensional settings.
We propose two bounded comparison metrics that may be implemented to arbitrary dimensions in regression tasks. One quantifies the structure of uncertainty and the other quantifies the distribution of uncertainty. The structure metric assesses the similarity in shape and location of uncertainty with the true error, while the distribution metric quantifies the supported magnitudes between the two. We apply these metrics to Gaussian Processes (GPs), Ensemble Deep Neural Nets (DNNs), and Ensemble Deep Neural Operators (DNOs) on high-dimensional and nonlinear test cases. We find that comparing a model's uncertainty estimates with the model's squared error provides a compelling ground truth assessment. We also observe that both DNNs and DNOs, especially when compared to GPs, provide encouraging metric values in high dimensions with either sparse or plentiful data.