Asymptotic Consistency of $α-$Rényi-Approximate Posteriors
This work addresses theoretical guarantees for variational Bayesian methods, which is important for researchers in Bayesian statistics and machine learning, but it is incremental as it builds on existing frequentist analyses of such methods.
The paper tackles the asymptotic consistency of α-Rényi approximate posteriors in variational Bayesian methods, particularly for α > 1, and identifies sufficient conditions for consistency, including the existence of a 'good' sequence of distributions with the right convergence rate.
We study the asymptotic consistency properties of $α$-Rényi approximate posteriors, a class of variational Bayesian methods that approximate an intractable Bayesian posterior with a member of a tractable family of distributions, the member chosen to minimize the $α$-Rényi divergence from the true posterior. Unique to our work is that we consider settings with $α> 1$, resulting in approximations that upperbound the log-likelihood, and consequently have wider spread than traditional variational approaches that minimize the Kullback-Liebler (KL) divergence from the posterior. Our primary result identifies sufficient conditions under which consistency holds, centering around the existence of a 'good' sequence of distributions in the approximating family that possesses, among other properties, the right rate of convergence to a limit distribution. We further characterize the good sequence by demonstrating that a sequence of distributions that converges too quickly cannot be a good sequence. We also extend our analysis to the setting where $α$ equals one, corresponding to the minimizer of the reverse KL divergence, and to models with local latent variables. We also illustrate the existence of good sequence with a number of examples. Our results complement a growing body of work focused on the frequentist properties of variational Bayesian methods.