Local Max-Entropy and Free Energy Principles, Belief Diffusions and their Singularities
This work provides a theoretical framework for understanding belief propagation and its singularities, which is incremental for researchers in statistical physics and machine learning.
The paper tackles the generalization of belief propagation equations to continuous-time diffusions for solving localized max-entropy and free energy principles on hypergraphs, showing that critical points and stationary beliefs arise from intersections of constraint surfaces and describing singularities via polynomial equations.
A comprehensive picture of three Bethe-Kikuchi variational principles including their relationship to belief propagation (BP) algorithms on hypergraphs is given. The structure of BP equations is generalized to define continuous-time diffusions, solving localized versions of the max-entropy principle (A), the variational free energy principle (B), and a less usual equilibrium free energy principle (C), Legendre dual to A. Both critical points of Bethe-Kikuchi functionals and stationary beliefs are shown to lie at the non-linear intersection of two constraint surfaces, enforcing energy conservation and marginal consistency respectively. The hypersurface of singular beliefs, accross which equilibria become unstable as the constraint surfaces meet tangentially, is described by polynomial equations in the convex polytope of consistent beliefs. This polynomial is expressed by a loop series expansion for graphs of binary variables.