Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent
This work provides a theoretical framework for understanding learning dynamics in deep learning, with implications for techniques like normalization and warmup, but it is incremental as it builds on existing symmetry concepts.
The paper tackles how exponential symmetries in neural networks interact with stochastic gradient descent (SGD), proving that gradient noise drives parameters to a unique fixed point called noise equilibria, and shows this mechanism explains phenomena like progressive sharpening and representation formation.
Symmetries are prevalent in deep learning and can significantly influence the learning dynamics of neural networks. In this paper, we examine how exponential symmetries -- a broad subclass of continuous symmetries present in the model architecture or loss function -- interplay with stochastic gradient descent (SGD). We first prove that gradient noise creates a systematic motion (a ``Noether flow") of the parameters $θ$ along the degenerate direction to a unique initialization-independent fixed point $θ^*$. These points are referred to as the {\it noise equilibria} because, at these points, noise contributions from different directions are balanced and aligned. Then, we show that the balance and alignment of gradient noise can serve as a novel alternative mechanism for explaining important phenomena such as progressive sharpening/flattening and representation formation within neural networks and have practical implications for understanding techniques like representation normalization and warmup.