Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence
This addresses uncertainty quantification for deploying deep learning in safety-critical areas, offering a post-hoc method that avoids sample splitting inefficiencies, though it is incremental as it builds on existing conformal prediction techniques.
The paper tackles the problem of constructing well-calibrated and sharp prediction intervals for neural network regressors without held-out data, by approximating full conformal prediction using Gauss-Newton influence and linearization, resulting in locally-adaptive intervals that are often tighter than split conformal prediction on benchmarks.
Uncertainty quantification is an important prerequisite for the deployment of deep learning models in safety-critical areas. Yet, this hinges on the uncertainty estimates being useful to the extent the prediction intervals are well-calibrated and sharp. In the absence of inherent uncertainty estimates (e.g. pretrained models predicting only point estimates), popular approaches that operate post-hoc include Laplace's method and split conformal prediction (split-CP). However, Laplace's method can be miscalibrated when the model is misspecified and split-CP requires sample splitting, and thus comes at the expense of statistical efficiency. In this work, we construct prediction intervals for neural network regressors post-hoc without held-out data. This is achieved by approximating the full conformal prediction method (full-CP). Whilst full-CP nominally requires retraining the model for every test point and candidate label, we propose to train just once and locally perturb model parameters using Gauss-Newton influence to approximate the effect of retraining. Coupled with linearization of the network, we express the absolute residual nonconformity score as a piecewise linear function of the candidate label allowing for an efficient procedure that avoids the exhaustive search over the output space. On standard regression benchmarks and bounding box localization, we show the resulting prediction intervals are locally-adaptive and often tighter than those of split-CP.