Symmetry-via-Duality: Invariant Neural Network Densities from Parameter-Space Correlators
This work addresses the challenge of understanding symmetry in neural network densities for researchers in machine learning theory, but it is incremental as it builds on known duality concepts and applies them to specific cases like initialization and training.
The paper tackled the problem of determining symmetries in neural network densities by leveraging duality between parameter-space and function-space, showing that symmetry in initialization affects accuracy on Fashion-MNIST and symmetry breaking helps only when aligned with ground truth, with concrete results on dataset performance.
Parameter-space and function-space provide two different duality frames in which to study neural networks. We demonstrate that symmetries of network densities may be determined via dual computations of network correlation functions, even when the density is unknown and the network is not equivariant. Symmetry-via-duality relies on invariance properties of the correlation functions, which stem from the choice of network parameter distributions. Input and output symmetries of neural network densities are determined, which recover known Gaussian process results in the infinite width limit. The mechanism may also be utilized to determine symmetries during training, when parameters are correlated, as well as symmetries of the Neural Tangent Kernel. We demonstrate that the amount of symmetry in the initialization density affects the accuracy of networks trained on Fashion-MNIST, and that symmetry breaking helps only when it is in the direction of ground truth.