Stochastic complexity of Bayesian networks
This work addresses the lack of mathematical foundation for learning in Bayesian networks, which are widely used in fields like diagnosis and data mining, by providing insights that could improve model selection criteria like BIC, though it appears incremental as it builds on prior methods.
The paper tackles the problem of analyzing the statistical properties of Bayesian networks with latent variables, which are non-regular and nonidentifiable, by applying an algebraic geometry method to clarify the order of stochastic complexities, finding that the upper bound is smaller than the parameter space dimension.
Bayesian networks are now being used in enormous fields, for example, diagnosis of a system, data mining, clustering and so on. In spite of their wide range of applications, the statistical properties have not yet been clarified, because the models are nonidentifiable and non-regular. In a Bayesian network, the set of its parameter for a smaller model is an analytic set with singularities in the space of large ones. Because of these singularities, the Fisher information matrices are not positive definite. In other words, the mathematical foundation for learning was not constructed. In recent years, however, we have developed a method to analyze non-regular models using algebraic geometry. This method revealed the relation between the models singularities and its statistical properties. In this paper, applying this method to Bayesian networks with latent variables, we clarify the order of the stochastic complexities.Our result claims that the upper bound of those is smaller than the dimension of the parameter space. This means that the Bayesian generalization error is also far smaller than that of regular model, and that Schwarzs model selection criterion BIC needs to be improved for Bayesian networks.