Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
This work provides a theoretical framework for understanding the learning dynamics of deep neural networks, which is crucial for researchers and practitioners aiming to build a more robust theoretical foundation for deep learning.
This paper tackles the challenge of understanding neural network parameter dynamics during training by introducing a theoretical framework based on intrinsic architectural symmetries. It shows that these symmetries impose geometric constraints on gradients and Hessians, leading to conservation laws in the continuous-time limit of SGD, and further demonstrates that finite learning rates break these laws. The authors derive modified gradient flow and exact integral expressions for parameter dynamics, which are empirically validated on VGG-16 trained on Tiny ImageNet.
Understanding the dynamics of neural network parameters during training is one of the key challenges in building a theoretical foundation for deep learning. A central obstacle is that the motion of a network in high-dimensional parameter space undergoes discrete finite steps along complex stochastic gradients derived from real-world datasets. We circumvent this obstacle through a unifying theoretical framework based on intrinsic symmetries embedded in a network's architecture that are present for any dataset. We show that any such symmetry imposes stringent geometric constraints on gradients and Hessians, leading to an associated conservation law in the continuous-time limit of stochastic gradient descent (SGD), akin to Noether's theorem in physics. We further show that finite learning rates used in practice can actually break these symmetry induced conservation laws. We apply tools from finite difference methods to derive modified gradient flow, a differential equation that better approximates the numerical trajectory taken by SGD at finite learning rates. We combine modified gradient flow with our framework of symmetries to derive exact integral expressions for the dynamics of certain parameter combinations. We empirically validate our analytic expressions for learning dynamics on VGG-16 trained on Tiny ImageNet. Overall, by exploiting symmetry, our work demonstrates that we can analytically describe the learning dynamics of various parameter combinations at finite learning rates and batch sizes for state of the art architectures trained on any dataset.