An Overview of Uncertainty Quantification Methods for Infinite Neural Networks
This is an incremental review that synthesizes existing theoretical tools for researchers studying large neural networks.
The paper reviews methods for quantifying uncertainty in infinite-width neural networks, leveraging equivalence results to derive exact closed-form solutions for predictive uncertainty.
To better understand the theoretical behavior of large neural networks, several works have analyzed the case where a network's width tends to infinity. In this regime, the effect of random initialization and the process of training a neural network can be formally expressed with analytical tools like Gaussian processes and neural tangent kernels. In this paper, we review methods for quantifying uncertainty in such infinite-width neural networks and compare their relationship to Gaussian processes in the Bayesian inference framework. We make use of several equivalence results along the way to obtain exact closed-form solutions for predictive uncertainty.