Laws of thermodynamics for exponential families
This work provides a foundational framework for understanding learning in terms of thermodynamics, potentially impacting all of ML/AI.
The paper tackles the problem of connecting thermodynamics to learning by developing laws of thermodynamics for exponential families, translating concepts like work and heat to AI/statistics terms, with implications for quantifying distribution shift.
We develop the laws of thermodynamics in terms of general exponential families. By casting learning (log-loss minimization) problems in max-entropy and statistical mechanics terms, we translate thermodynamics results to learning scenarios. We extend the well-known way in which exponential families characterize thermodynamic and learning equilibria. Basic ideas of work and heat, and advanced concepts of thermodynamic cycles and equipartition of energy, find exact and useful counterparts in AI / statistics terms. These ideas have broad implications for quantifying and addressing distribution shift.