A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networks
For researchers in deep learning theory, this work provides a novel mathematical framework to analyze training dynamics by removing symmetry redundancies, potentially offering principled diagnostics for monitoring and controlling training.
The paper establishes a mathematical link between shock-wave theory and symmetry-reduced stochastic gradient descent, showing that after quotienting symmetries and coarse-graining, the dynamics satisfy viscous Hamilton--Jacobi and Burgers-type equations. This framework is applied to various architectures including Transformers, and is conjectured to provide practical diagnostics for training-phase transitions.
We develop a mathematically explicit link between shock-wave theory and the symmetry-quotiented learning dynamics of stochastic gradient descent, drawing on differential geometry, Lie group theory, and fluid mechanics. Specifically, after quotienting parameter symmetries and applying local-entropy coarse-graining, the effective dynamics satisfy a viscous Hamilton--Jacobi equation on the quotient manifold. Moreover, under the assumption that the raw parameter dynamics can be summarized by a gradient field on the quotiented space, the gradient of the coarse-grained loss function obeys a Burgers-type equation, and shock formation can be established rigorously. We apply our theory to multilayer perceptrons, convolutional neural networks, Transformers, and mean-field networks, and show that they obey the Hamilton--Jacobi or Burgers-type equations. We conjecture that this framework also yields practical diagnostics for deep learning. In architectures such as Transformers, raw parameter norms are often distorted by symmetry redundancy and may therefore be misleading, whereas symmetry-corrected quotient observables provide a principled basis for monitoring, forecasting, and controlling training-phase transitions.