Thermodynamics of Reinforcement Learning Curricula
This work addresses curriculum design for reinforcement learning practitioners, offering a principled approach based on thermodynamics, though it appears incremental in applying existing thermodynamic concepts to RL.
The paper tackles the problem of curriculum learning in reinforcement learning by formalizing it using non-equilibrium thermodynamics, showing that optimal curricula correspond to geodesics on a task manifold and resulting in the development of the MEW algorithm for temperature annealing in maximum-entropy RL.
Connections between statistical mechanics and machine learning have repeatedly proven fruitful, providing insight into optimization, generalization, and representation learning. In this work, we follow this tradition by leveraging results from non-equilibrium thermodynamics to formalize curriculum learning in reinforcement learning (RL). In particular, we propose a geometric framework for RL by interpreting reward parameters as coordinates on a task manifold. We show that, by minimizing the excess thermodynamic work, optimal curricula correspond to geodesics in this task space. As an application of this framework, we provide an algorithm, "MEW" (Minimum Excess Work), to derive a principled schedule for temperature annealing in maximum-entropy RL.