Rebuilding Factorized Information Criterion: Asymptotically Accurate Marginal Likelihood
This work improves model selection for latent variable models, offering a more rigorous and applicable framework, though it is incremental as it builds on existing FIC studies.
The paper addresses theoretical gaps in the factorized information criterion (FIC) by generalizing it to a broader class of latent variable models, including continuous ones, and provides formal justification and a systematic pruning procedure for model selection, supported by numerical experiments.
Factorized information criterion (FIC) is a recently developed approximation technique for the marginal log-likelihood, which provides an automatic model selection framework for a few latent variable models (LVMs) with tractable inference algorithms. This paper reconsiders FIC and fills theoretical gaps of previous FIC studies. First, we reveal the core idea of FIC that allows generalization for a broader class of LVMs, including continuous LVMs, in contrast to previous FICs, which are applicable only to binary LVMs. Second, we investigate the model selection mechanism of the generalized FIC. Our analysis provides a formal justification of FIC as a model selection criterion for LVMs and also a systematic procedure for pruning redundant latent variables that have been removed heuristically in previous studies. Third, we provide an interpretation of FIC as a variational free energy and uncover a few previously-unknown their relationships. A demonstrative study on Bayesian principal component analysis is provided and numerical experiments support our theoretical results.