Categorical Invariants of Learning Dynamics
This work provides theoretical insight and practical tools for training more robust networks, addressing a foundational problem in machine learning, though it is incremental in applying categorical methods to learning dynamics.
The paper tackles the problem of understanding neural network training dynamics by proposing a categorical framework that views learning as a structure-preserving transformation, revealing that training runs with similar test performance often belong to the same homotopy class, with experiments showing networks in homotopic trajectories generalize within 0.5% accuracy while non-homotopic paths differ by over 3%.
Neural network training is typically viewed as gradient descent on a loss surface. We propose a fundamentally different perspective: learning is a structure-preserving transformation (a functor L) between the space of network parameters (Param) and the space of learned representations (Rep). This categorical framework reveals that different training runs producing similar test performance often belong to the same homotopy class (continuous deformation family) of optimization paths. We show experimentally that networks converging via homotopic trajectories generalize within 0.5% accuracy of each other, while non-homotopic paths differ by over 3%. The theory provides practical tools: persistent homology identifies stable minima predictive of generalization (R^2 = 0.82 correlation), pullback constructions formalize transfer learning, and 2-categorical structures explain when different optimization algorithms yield functionally equivalent models. These categorical invariants offer both theoretical insight into why deep learning works and concrete algorithmic principles for training more robust networks.