Mean-field variational inference with the TAP free energy: Geometric and statistical properties in linear models
This addresses the challenge of accurate posterior inference in high-dimensional statistics, offering a method that improves over standard variational approaches, though it is incremental in refining existing techniques.
The paper tackles the problem of mean-field variational inference in high-dimensional Bayesian linear models, showing that minimizing the TAP free energy yields a consistent estimate of posterior marginals and correctly calibrated inference, with an efficient algorithm that converges linearly to the minimizer.
We study mean-field variational inference in a Bayesian linear model when the sample size n is comparable to the dimension p. In high dimensions, the common approach of minimizing a Kullback-Leibler divergence from the posterior distribution, or maximizing an evidence lower bound, may deviate from the true posterior mean and underestimate posterior uncertainty. We study instead minimization of the TAP free energy, showing in a high-dimensional asymptotic framework that it has a local minimizer which provides a consistent estimate of the posterior marginals and may be used for correctly calibrated posterior inference. Geometrically, we show that the landscape of the TAP free energy is strongly convex in an extensive neighborhood of this local minimizer, which under certain general conditions can be found by an Approximate Message Passing (AMP) algorithm. We then exhibit an efficient algorithm that linearly converges to the minimizer within this local neighborhood. In settings where it is conjectured that no efficient algorithm can find this local neighborhood, we prove analogous geometric properties for a local minimizer of the TAP free energy reachable by AMP, and show that posterior inference based on this minimizer remains correctly calibrated.