Sharpness of Minima in Deep Matrix Factorization: Exact Expressions
This provides theoretical insights into the geometry of loss landscapes for researchers in non-convex optimization, though it is incremental as it builds on existing frameworks.
The paper tackles the problem of characterizing the sharpness of minima in deep matrix factorization by deriving the first exact expression for the maximum eigenvalue of the Hessian at any minimizer, resolving an open question from prior work.
Understanding the geometry of the loss landscape near a minimum is key to explaining the implicit bias of gradient-based methods in non-convex optimization problems such as deep neural network training and deep matrix factorization. A central quantity to characterize this geometry is the maximum eigenvalue of the Hessian of the loss, which measures the sharpness of the landscape. Currently, its precise role has been obfuscated because no exact expressions for this sharpness measure were known in general settings. In this paper, we present the first exact expression for the maximum eigenvalue of the Hessian of the squared-error loss at any minimizer in general overparameterized deep matrix factorization (i.e., deep linear neural network training) problems, resolving an open question posed by Mulayoff & Michaeli (2020). To complement our theory, we empirically investigate an escape phenomenon observed during gradient-based training near a minimum that crucially relies on our exact expression of the sharpness.