Gradient Span Algorithms Make Predictable Progress in High Dimension
This addresses the problem of understanding training consistency in high-dimensional non-convex optimization for machine learning researchers, providing a theoretical explanation for observed empirical phenomena.
The paper proves that gradient span algorithms exhibit asymptotically deterministic behavior on scaled Gaussian random functions in high dimensions, explaining why different training runs of large machine learning models yield similar cost curves despite random initialization on non-convex landscapes.
We prove that all 'gradient span algorithms' have asymptotically deterministic behavior on scaled Gaussian random functions as the dimension tends to infinity. In particular, this result explains the counterintuitive phenomenon that different training runs of many large machine learning models result in approximately equal cost curves despite random initialization on a complicated non-convex landscape. The distributional assumption of (non-stationary) isotropic Gaussian random functions we use is sufficiently general to serve as realistic model for machine learning training but also encompass spin glasses and random quadratic functions.