Symmetry Induces Structure and Constraint of Learning
This work addresses the problem of understanding and controlling learning behavior in neural networks for researchers, offering a theoretical framework to explain phenomena like loss of plasticity and collapse, though it is incremental in building on known symmetry concepts.
The paper demonstrates that symmetries in loss functions impose constraints on model parameters, leading to sparsity, low rankness, and homogeneous ensembling, with proven results under conditions like large weight decay or gradient noise.
Due to common architecture designs, symmetries exist extensively in contemporary neural networks. In this work, we unveil the importance of the loss function symmetries in affecting, if not deciding, the learning behavior of machine learning models. We prove that every mirror-reflection symmetry, with reflection surface $O$, in the loss function leads to the emergence of a constraint on the model parameters $θ$: $O^Tθ=0$. This constrained solution becomes satisfied when either the weight decay or gradient noise is large. Common instances of mirror symmetries in deep learning include rescaling, rotation, and permutation symmetry. As direct corollaries, we show that rescaling symmetry leads to sparsity, rotation symmetry leads to low rankness, and permutation symmetry leads to homogeneous ensembling. Then, we show that the theoretical framework can explain intriguing phenomena, such as the loss of plasticity and various collapse phenomena in neural networks, and suggest how symmetries can be used to design an elegant algorithm to enforce hard constraints in a differentiable way.