Sandwiching the marginal likelihood using bidirectional Monte Carlo
This work addresses the challenge of measuring accuracy in ML estimates for researchers in statistics and machine learning, though it is incremental as it builds on existing methods like annealed importance sampling.
The paper tackled the problem of accurately estimating the marginal likelihood (ML) in latent variable models by introducing bidirectional Monte Carlo, which provides stochastic lower and upper bounds to sandwich the true value with high probability, enabling quantitative evaluation of existing ML estimators on models like clustering and low-rank approximations.
Computing the marginal likelihood (ML) of a model requires marginalizing out all of the parameters and latent variables, a difficult high-dimensional summation or integration problem. To make matters worse, it is often hard to measure the accuracy of one's ML estimates. We present bidirectional Monte Carlo, a technique for obtaining accurate log-ML estimates on data simulated from a model. This method obtains stochastic lower bounds on the log-ML using annealed importance sampling or sequential Monte Carlo, and obtains stochastic upper bounds by running these same algorithms in reverse starting from an exact posterior sample. The true value can be sandwiched between these two stochastic bounds with high probability. Using the ground truth log-ML estimates obtained from our method, we quantitatively evaluate a wide variety of existing ML estimators on several latent variable models: clustering, a low rank approximation, and a binary attributes model. These experiments yield insights into how to accurately estimate marginal likelihoods.