Free Energy Minimization: A Unified Framework for Modelling, Inference, Learning,and Optimization
This paper provides a conceptual unification for researchers and practitioners working across different areas of machine learning and optimization by showing how free energy minimization underpins various techniques.
This paper reviews free energy minimization as a unified framework for various machine learning and optimization problems. It covers maximum entropy modelling, generalized Bayesian inference, learning with latent variables, statistical learning analysis of generalization, and local optimization, starting from its thermodynamic origins and mathematical description via Fenchel duality.
The goal of these lecture notes is to review the problem of free energy minimization as a unified framework underlying the definition of maximum entropy modelling, generalized Bayesian inference, learning with latent variables, statistical learning analysis of generalization,and local optimization. Free energy minimization is first introduced, here and historically, as a thermodynamic principle. Then, it is described mathematically in the context of Fenchel duality. Finally, the mentioned applications to modelling, inference, learning, and optimization are covered starting from basic principles.