Critical Points of Degenerate Metrics on Algebraic Varieties: A Tale of Overparametrization
This work addresses theoretical challenges in understanding optimization landscapes for overparametrized models, which is relevant for machine learning practitioners dealing with small datasets relative to model size.
The paper tackles the problem of finding critical points in degenerate optimization problems that arise in overparametrized machine learning models, relating them to nondegenerate problems via projection and showing that the ramification locus plays a central role in highly-degenerate regimes.
We study the critical points over an algebraic variety of an optimization problem defined by a quadratic objective that is degenerate. This scenario arises in machine learning when the dataset size is small with respect to the model, and is typically referred to as overparametrization. Our main result relates the degenerate optimization problem to a nondegenerate one via a projection. In the highly-degenerate regime, we find that a central role is played by the ramification locus of the projection. Additionally, we provide tools for counting the number of critical points over projective varieties, and discuss specific cases arising from deep learning. Our work bridges tools from algebraic geometry with ideas from machine learning, and it extends the line of literature around the Euclidean distance degree to the degenerate setting.