Ubiquitous Symmetry at Critical Points Across Diverse Optimization Landscapes
This work addresses the fundamental problem of understanding symmetry in optimization for researchers in mathematics and machine learning, but it is incremental as it extends prior findings to new cases.
The paper investigates symmetry at critical points in optimization landscapes beyond neural networks, examining four additional cases and finding that all observed critical points exhibit non-trivial symmetry. It also introduces a new symmetry measure that reveals additional structures not captured by previous measures.
Symmetry plays a crucial role in understanding the properties of mathematical structures and optimization problems. Recent work has explored this phenomenon in the context of neural networks, where the loss function is invariant under column and row permutations of the network weights. It has been observed that local minima exhibit significant symmetry with respect to the network weights (invariance to row and column permutations). And moreover no critical point was found that lacked symmetry. We extend this line of inquiry by investigating symmetry phenomena in real-valued loss functions defined on a broader class of spaces. We will introduce four more cases: the projective case over a finite field, the octahedral graph case, the perfect matching case, and the particle attraction case. We show that as in the neural network case, all the critical points observed have non-trivial symmetry. Finally we introduce a new measure of symmetry in the system and show that it reveals additional symmetry structures not captured by the previous measure.