The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability
This provides a theoretical foundation for understanding the mechanisms of generative diffusion models, which are widely used in machine learning, but it is incremental as it builds on existing physics concepts.
The paper reformulates generative diffusion models using equilibrium statistical mechanics, revealing that they undergo second-order phase transitions with mean-field universality, which underlies their generative capabilities through critical instability and specific critical exponents.
Generative diffusion models have achieved spectacular performance in many areas of machine learning and generative modeling. While the fundamental ideas behind these models come from non-equilibrium physics, variational inference and stochastic calculus, in this paper we show that many aspects of these models can be understood using the tools of equilibrium statistical mechanics. Using this reformulation, we show that generative diffusion models undergo second-order phase transitions corresponding to symmetry breaking phenomena. We show that these phase-transitions are always in a mean-field universality class, as they are the result of a self-consistency condition in the generative dynamics. We argue that the critical instability that arises from the phase transitions lies at the heart of their generative capabilities, which are characterized by a set of mean-field critical exponents. Finally, we show that the dynamic equation of the generative process can be interpreted as a stochastic adiabatic transformation that minimizes the free energy while keeping the system in thermal equilibrium.