Improving Output Uncertainty Estimation and Generalization in Deep Learning via Neural Network Gaussian Processes
This addresses uncertainty quantification and generalization issues for practitioners using deep learning models, though it appears incremental as it hybridizes existing techniques.
The authors tackled the problem of improving uncertainty estimation and generalization in deep learning by combining neural networks and Gaussian processes, achieving better performance than either method alone on real-world spatio-temporal datasets.
We propose a simple method that combines neural networks and Gaussian processes. The proposed method can estimate the uncertainty of outputs and flexibly adjust target functions where training data exist, which are advantages of Gaussian processes. The proposed method can also achieve high generalization performance for unseen input configurations, which is an advantage of neural networks. With the proposed method, neural networks are used for the mean functions of Gaussian processes. We present a scalable stochastic inference procedure, where sparse Gaussian processes are inferred by stochastic variational inference, and the parameters of neural networks and kernels are estimated by stochastic gradient descent methods, simultaneously. We use two real-world spatio-temporal data sets to demonstrate experimentally that the proposed method achieves better uncertainty estimation and generalization performance than neural networks and Gaussian processes.