Fixed-Mean Gaussian Processes for Post-hoc Bayesian Deep Learning
This addresses the need for reliable uncertainty measures in deep learning applications without retraining models, though it is incremental as it builds on existing sparse variational GP methods.
The paper tackles the problem of post-hoc uncertainty estimation for pre-trained deep neural networks by introducing fixed-mean Gaussian processes, which fix the posterior mean to the DNN output to fit predictive variances, resulting in improved uncertainty estimation and computational efficiency compared to state-of-the-art methods.
Recently, there has been an increasing interest in performing post-hoc uncertainty estimation about the predictions of pre-trained deep neural networks (DNNs). Given a pre-trained DNN via back-propagation, these methods enhance the original network by adding output confidence measures, such as error bars, without compromising its initial accuracy. In this context, we introduce a novel family of sparse variational Gaussian processes (GPs), where the posterior mean is fixed to any continuous function when using a universal kernel. Specifically, we fix the mean of this GP to the output of the pre-trained DNN, allowing our approach to effectively fit the GP's predictive variances to estimate the DNN prediction uncertainty. Our approach leverages variational inference (VI) for efficient stochastic optimization, with training costs that remain independent of the number of training points, scaling efficiently to large datasets such as ImageNet. The proposed method, called fixed mean GP (FMGP), is architecture-agnostic, relying solely on the pre-trained model's outputs to adjust the predictive variances. Experimental results demonstrate that FMGP improves both uncertainty estimation and computational efficiency when compared to state-of-the-art methods.