Accuracy-Reliability Cost Function for Empirical Variance Estimation
This addresses the need for reliable uncertainty estimation in heteroskedastic regression problems, but it is incremental as it builds on existing probabilistic forecasting methods.
The paper tackles the problem of assigning uncertainties to single-point predictions by introducing an Accuracy-Reliability cost function that balances accuracy and reliability in probabilistic forecasts, showing examples where the underlying noise function is accurately recovered in synthetic data.
In this paper we focus on the problem of assigning uncertainties to single-point predictions. We introduce a cost function that encodes the trade-off between accuracy and reliability in probabilistic forecast. We derive analytic formula for the case of forecasts of continuous scalar variables expressed in terms of Gaussian distributions. The Accuracy-Reliability cost function can be used to empirically estimate the variance in heteroskedastic regression problems (input dependent noise), by solving a two-objective optimization problem. The simple philosophy behind this strategy is that predictions based on the estimated variances should be both accurate and reliable (i.e. statistical consistent with observations). We show several examples with synthetic data, where the underlying hidden noise function can be accurately recovered, both in one and multi-dimensional problems. The practical implementation of the method has been done using a Neural Network and, in the one-dimensional case, with a simple polynomial fit.