Convergence Rates of Empirical Bayes Posterior Distributions: A Variational Perspective
This work provides theoretical foundations for empirical Bayes methods, which is important for statisticians and machine learning practitioners dealing with uncertainty quantification in complex models.
The paper tackles the problem of establishing convergence rates for empirical Bayes posterior distributions in nonparametric and high-dimensional inference, showing that these rates can be derived from a variational perspective and applying the theory to statistical estimation problems like density estimation and regression.
We study the convergence rates of empirical Bayes posterior distributions for nonparametric and high-dimensional inference. We show that as long as the hyperparameter set is discrete, the empirical Bayes posterior distribution induced by the maximum marginal likelihood estimator can be regarded as a variational approximation to a hierarchical Bayes posterior distribution. This connection between empirical Bayes and variational Bayes allows us to leverage the recent results in the variational Bayes literature, and directly obtains the convergence rates of empirical Bayes posterior distributions from a variational perspective. For a more general hyperparameter set that is not necessarily discrete, we introduce a new technique called "prior decomposition" to deal with prior distributions that can be written as convex combinations of probability measures whose supports are low-dimensional subspaces. This leads to generalized versions of the classical "prior mass and testing" conditions for the convergence rates of empirical Bayes. Our theory is applied to a number of statistical estimation problems including nonparametric density estimation and sparse linear regression.