Geometry Perspective Of Estimating Learning Capability Of Neural Networks
This work provides a theoretical framework for understanding learning dynamics in neural networks, which is incremental as it builds on existing physics-inspired concepts.
The paper tackles the problem of estimating neural network learning capability by analyzing system characteristics at critical training epochs, proving that higher generalization capability leads to slower convergence rates and establishing a variant of the Complexity-Action conjecture from a neural network perspective.
The paper uses statistical and differential geometric motivation to acquire prior information about the learning capability of an artificial neural network on a given dataset. The paper considers a broad class of neural networks with generalized architecture performing simple least square regression with stochastic gradient descent (SGD). The system characteristics at two critical epochs in the learning trajectory are analyzed. During some epochs of the training phase, the system reaches equilibrium with the generalization capability attaining a maximum. The system can also be coherent with localized, non-equilibrium states, which is characterized by the stabilization of the Hessian matrix. The paper proves that neural networks with higher generalization capability will have a slower convergence rate. The relationship between the generalization capability with the stability of the neural network has also been discussed. By correlating the principles of high-energy physics with the learning theory of neural networks, the paper establishes a variant of the Complexity-Action conjecture from an artificial neural network perspective.