Neural Thermodynamic Laws for Large Language Model Training
This work addresses a foundational problem in machine learning by offering new theoretical insights into LLM training dynamics, though it appears incremental as it builds on existing scaling laws.
The paper tackles the lack of known laws for large language model (LLM) training by introducing Neural Thermodynamic Laws (NTL), a framework that shows thermodynamic principles emerge from loss landscapes and provides guidelines for learning rate schedules.
Beyond neural scaling laws, little is known about the laws underlying large language models (LLMs). We introduce Neural Thermodynamic Laws (NTL) -- a new framework that offers fresh insights into LLM training dynamics. On the theoretical side, we demonstrate that key thermodynamic quantities (e.g., temperature, entropy, heat capacity, thermal conduction) and classical thermodynamic principles (e.g., the three laws of thermodynamics and the equipartition theorem) naturally emerge under river-valley loss landscape assumptions. On the practical side, this scientific perspective yields intuitive guidelines for designing learning rate schedules.