Optimization Dynamics of Equivariant and Augmented Neural Networks
This work addresses optimization challenges in machine learning for symmetric data, providing theoretical insights that are incremental to existing equivariance and augmentation methods.
The paper investigates optimization dynamics for neural networks on symmetric data, comparing equivariant architectures with data augmentation, and finds that under certain conditions, both strategies share identical sets of equivariant stationary points, but augmented training can lead to unstable stationary points even when equivariant models are stable.
We investigate the optimization of neural networks on symmetric data, and compare the strategy of constraining the architecture to be equivariant to that of using data augmentation. Our analysis reveals that that the relative geometry of the admissible and the equivariant layers, respectively, plays a key role. Under natural assumptions on the data, network, loss, and group of symmetries, we show that compatibility of the spaces of admissible layers and equivariant layers, in the sense that the corresponding orthogonal projections commute, implies that the sets of equivariant stationary points are identical for the two strategies. If the linear layers of the network also are given a unitary parametrization, the set of equivariant layers is even invariant under the gradient flow for augmented models. Our analysis however also reveals that even in the latter situation, stationary points may be unstable for augmented training although they are stable for the manifestly equivariant models.