NGD converges to less degenerate solutions than SGD
This addresses the problem of understanding optimization dynamics in machine learning for researchers, though it appears incremental as it compares existing methods on known theoretical concepts.
The paper investigates whether natural gradient descent (NGD) leads to less degenerate solutions than stochastic gradient descent (SGD) by comparing their effective dimensions using singular learning theory metrics, finding that NGD-trained models consistently have higher effective dimensions.
The number of free parameters, or dimension, of a model is a straightforward way to measure its complexity: a model with more parameters can encode more information. However, this is not an accurate measure of complexity: models capable of memorizing their training data often generalize well despite their high dimension. Effective dimension aims to more directly capture the complexity of a model by counting only the number of parameters required to represent the functionality of the model. Singular learning theory (SLT) proposes the learning coefficient $ λ$ as a more accurate measure of effective dimension. By describing the rate of increase of the volume of the region of parameter space around a local minimum with respect to loss, $ λ$ incorporates information from higher-order terms. We compare $ λ$ of models trained using natural gradient descent (NGD) and stochastic gradient descent (SGD), and find that those trained with NGD consistently have a higher effective dimension for both of our methods: the Hessian trace $ \text{Tr}(\mathbf{H}) $, and the estimate of the local learning coefficient (LLC) $ \hatλ(w^*) $.