Symmetry in Neural Network Parameter Spaces
This is a survey paper that synthesizes existing literature on symmetry in neural networks, identifying connections and gaps, but it is incremental as it does not present new experimental results or methods.
The paper surveys parameter space symmetries in overparameterized neural networks, which explain redundancy in parameter configurations and influence the loss landscape and learning dynamics, offering a new perspective for understanding optimization, generalization, and model complexity.
Modern deep learning models are highly overparameterized, resulting in large sets of parameter configurations that yield the same outputs. A significant portion of this redundancy is explained by symmetries in the parameter space--transformations that leave the network function unchanged. These symmetries shape the loss landscape and constrain learning dynamics, offering a new lens for understanding optimization, generalization, and model complexity that complements existing theory of deep learning. This survey provides an overview of parameter space symmetry. We summarize existing literature, uncover connections between symmetry and learning theory, and identify gaps and opportunities in this emerging field.