A deterministic and computable Bernstein-von Mises theorem
This provides a path towards rigorously controlling error in Laplace approximations for computational statistics, though it is incremental as it builds on classical asymptotic results.
The paper tackles the problem of making Bernstein-von Mises theorems computationally useful by presenting a new theorem that bounds the KL divergence between a fixed log-concave density and its Laplace approximation, with the bound going to zero as the density becomes more Gaussian, and empirical tests in logistic classification show the approximations are effective surrogates.
Bernstein-von Mises results (BvM) establish that the Laplace approximation is asymptotically correct in the large-data limit. However, these results are inappropriate for computational purposes since they only hold over most, and not all, datasets and involve hard-to-estimate constants. In this article, I present a new BvM theorem which bounds the Kullback-Leibler (KL) divergence between a fixed log-concave density $f\left(\boldsymbolθ\right)$ and its Laplace approximation. The bound goes to $0$ as the higher-derivatives of $f\left(\boldsymbolθ\right)$ tend to $0$ and $f\left(\boldsymbolθ\right)$ becomes increasingly Gaussian. The classical BvM theorem in the IID large-data asymptote is recovered as a corollary. Critically, this theorem further suggests a number of computable approximations of the KL divergence with the most promising being: \[ KL\left(g_{LAP},f\right)\approx\frac{1}{2}\text{Var}_{\boldsymbolθ\sim g\left(\boldsymbolθ\right)}\left(\log\left[f\left(\boldsymbolθ\right)\right]-\log\left[g_{LAP}\left(\boldsymbolθ\right)\right]\right) \] An empirical investigation of these bounds in the logistic classification model reveals that these approximations are great surrogates for the KL divergence. This result, and future results of a similar nature, could provide a path towards rigorously controlling the error due to the Laplace approximation and more modern approximation methods.