Topological Invariance and Breakdown in Learning
This provides a universal topological framework for analyzing deep learning dynamics, applicable across architectures and loss functions, but is incremental in linking to existing edge-of-stability phenomena.
The paper proves that for permutation-equivariant learning rules like SGD and Adam, training induces a bi-Lipschitz mapping that constrains neuron topology, with a critical learning rate η* separating phases where topology is preserved or simplified, reducing model expressivity.
We prove that for a broad class of permutation-equivariant learning rules (including SGD, Adam, and others), the training process induces a bi-Lipschitz mapping between neurons and strongly constrains the topology of the neuron distribution during training. This result reveals a qualitative difference between small and large learning rates $η$. With a learning rate below a topological critical point $η^*$, the training is constrained to preserve all topological structure of the neurons. In contrast, above $η^*$, the learning process allows for topological simplification, making the neuron manifold progressively coarser and thereby reducing the model's expressivity. Viewed in combination with the recent discovery of the edge of stability phenomenon, the learning dynamics of neuron networks under gradient descent can be divided into two phases: first they undergo smooth optimization under topological constraints, and then enter a second phase where they learn through drastic topological simplifications. A key feature of our theory is that it is independent of specific architectures or loss functions, enabling the universal application of topological methods to the study of deep learning.