Quantifying the probable approximation error of probabilistic inference programs
This work addresses the challenge of error quantification for practitioners using probabilistic inference, offering a method to assess approximation quality beyond predictive performance, though it is incremental in building on existing inference frameworks.
The paper tackles the problem of quantifying approximation error in probabilistic inference programs by introducing a technique that provides a subjective bound on the symmetrized KL divergence between approximate and true distributions, using empirical results from linear regression, Dirichlet process mixture modeling, HMMs, and Bayesian networks to show robustness and bug detection capabilities.
This paper introduces a new technique for quantifying the approximation error of a broad class of probabilistic inference programs, including ones based on both variational and Monte Carlo approaches. The key idea is to derive a subjective bound on the symmetrized KL divergence between the distribution achieved by an approximate inference program and its true target distribution. The bound's validity (and subjectivity) rests on the accuracy of two auxiliary probabilistic programs: (i) a "reference" inference program that defines a gold standard of accuracy and (ii) a "meta-inference" program that answers the question "what internal random choices did the original approximate inference program probably make given that it produced a particular result?" The paper includes empirical results on inference problems drawn from linear regression, Dirichlet process mixture modeling, HMMs, and Bayesian networks. The experiments show that the technique is robust to the quality of the reference inference program and that it can detect implementation bugs that are not apparent from predictive performance.