Inferring Parameters and Structure of Latent Variable Models by Variational Bayes
This provides a solution for researchers and practitioners in machine learning and statistics dealing with latent variable models, offering a more robust Bayesian approach compared to existing methods.
The paper tackles the problem of overfitting and suboptimal generalization in learning graphical models with latent variables by introducing the Variational Bayes framework, which approximates full posterior distributions over parameters, structures, and latent variables analytically without sampling, generalizing the Expectation Maximization algorithm with guaranteed convergence.
Current methods for learning graphical models with latent variables and a fixed structure estimate optimal values for the model parameters. Whereas this approach usually produces overfitting and suboptimal generalization performance, carrying out the Bayesian program of computing the full posterior distributions over the parameters remains a difficult problem. Moreover, learning the structure of models with latent variables, for which the Bayesian approach is crucial, is yet a harder problem. In this paper I present the Variational Bayes framework, which provides a solution to these problems. This approach approximates full posterior distributions over model parameters and structures, as well as latent variables, in an analytical manner without resorting to sampling methods. Unlike in the Laplace approximation, these posteriors are generally non-Gaussian and no Hessian needs to be computed. The resulting algorithm generalizes the standard Expectation Maximization algorithm, and its convergence is guaranteed. I demonstrate that this algorithm can be applied to a large class of models in several domains, including unsupervised clustering and blind source separation.