An Equivariance Toolbox for Learning Dynamics
This work addresses a foundational gap in deep learning theory for researchers, offering a unified framework that connects transformation structure to optimization geometry, though it is incremental in extending existing analyses.
The paper tackles the problem of understanding second-order structure in learning dynamics by developing a general equivariance toolbox that yields coupled first- and second-order constraints, extending classical analyses to include Hessian constraints, general equivariance, and discrete transformations, and providing structural predictions about curvature and loss landscape geometry.
Many theoretical results in deep learning can be traced to symmetry or equivariance of neural networks under parameter transformations. However, existing analyses are typically problem-specific and focus on first-order consequences such as conservation laws, while the implications for second-order structure remain less understood. We develop a general equivariance toolbox that yields coupled first- and second-order constraints on learning dynamics. The framework extends classical Noether-type analyses in three directions: from gradient constraints to Hessian constraints, from symmetry to general equivariance, and from continuous to discrete transformations. At the first order, our framework unifies conservation laws and implicit-bias relations as special cases of a single identity. At the second order, it provides structural predictions about curvature: which directions are flat or sharp, how the gradient aligns with Hessian eigenspaces, and how the loss landscape geometry reflects the underlying transformation structure. We illustrate the framework through several applications, recovering known results while also deriving new characterizations that connect transformation structure to modern empirical observations about optimization geometry.