A Monte Carlo Framework for Calibrated Uncertainty Estimation in Sequence Prediction
This work addresses uncertainty estimation for risk-sensitive applications like image-to-sequence tasks, but it is incremental as it builds on existing Monte Carlo and neural network methods.
The paper tackles the problem of quantifying uncertainty in sequence prediction from high-dimensional data, proposing a Monte Carlo framework that uses an autoregressive neural network to sample sequences and estimate probabilities and confidence intervals, with experiments showing accurate predictions but miscalibration, addressed by a time-dependent regularization method that achieves calibrated results.
Probabilistic prediction of sequences from images and other high-dimensional data is a key challenge, particularly in risk-sensitive applications. In these settings, it is often desirable to quantify the uncertainty associated with the prediction (instead of just determining the most likely sequence, as in language modeling). In this paper, we propose a Monte Carlo framework to estimate probabilities and confidence intervals associated with the distribution of a discrete sequence. Our framework uses a Monte Carlo simulator, implemented as an autoregressively trained neural network, to sample sequences conditioned on an image input. We then use these samples to estimate the probabilities and confidence intervals. Experiments on synthetic and real data show that the framework produces accurate discriminative predictions, but can suffer from miscalibration. In order to address this shortcoming, we propose a time-dependent regularization method, which is shown to produce calibrated predictions.