Practical Deep Heteroskedastic Regression
This addresses uncertainty quantification for critical applications like sequential decision making and risk-sensitive tasks, offering an incremental improvement in deep learning regression.
The paper tackles practical challenges in training deep heteroskedastic regression models, such as optimization difficulties and variance overfitting, by proposing a post-hoc fitting procedure that achieves on-par or state-of-the-art uncertainty quantification on molecular graph datasets without compromising mean prediction accuracy.
Uncertainty quantification (UQ) in deep learning regression is of wide interest, as it supports critical applications including sequential decision making and risk-sensitive tasks. In heteroskedastic regression, where the uncertainty of the target depends on the input, a common approach is to train a neural network that parameterizes the mean and the variance of the predictive distribution. Still, training deep heteroskedastic regression models poses practical challenges in the trade-off between uncertainty quantification and mean prediction, such as optimization difficulties, representation collapse, and variance overfitting. In this work we identify previously undiscussed fallacies and propose a simple and efficient procedure that addresses these challenges jointly by post-hoc fitting a variance model across the intermediate layers of a pretrained network on a hold-out dataset. We demonstrate that our method achieves on-par or state-of-the-art uncertainty quantification on several molecular graph datasets, without compromising mean prediction accuracy and remaining cheap to use at prediction time.