Asymptotics of Bayesian Uncertainty Estimation in Random Features Regression
This work provides theoretical insights into uncertainty quantification for practitioners in machine learning, though it is incremental as it builds on prior Gaussian sequence model results.
The paper analyzes the asymptotic behavior of Bayesian uncertainty estimation versus maximum a posteriori (MAP) estimator risk in overparameterized random features regression, finding agreement between them in regimes where model dimensions or samples grow faster than each other, governed by a signal-to-noise ratio phase transition.
In this paper we compare and contrast the behavior of the posterior predictive distribution to the risk of the maximum a posteriori estimator for the random features regression model in the overparameterized regime. We will focus on the variance of the posterior predictive distribution (Bayesian model average) and compare its asymptotics to that of the risk of the MAP estimator. In the regime where the model dimensions grow faster than any constant multiple of the number of samples, asymptotic agreement between these two quantities is governed by the phase transition in the signal-to-noise ratio. They also asymptotically agree with each other when the number of samples grow faster than any constant multiple of model dimensions. Numerical simulations illustrate finer distributional properties of the two quantities for finite dimensions. We conjecture they have Gaussian fluctuations and exhibit similar properties as found by previous authors in a Gaussian sequence model, which is of independent theoretical interest.