Two-Point Deterministic Equivalence for Stochastic Gradient Dynamics in Linear Models
This work provides a unified understanding of stochastic gradient descent performance for high-dimensional linear models, which is significant for machine learning researchers and practitioners working with these models.
The authors derived a novel deterministic equivalence for stochastic gradient dynamics in linear models, unifying the performance analysis of various high-dimensional linear models. This includes previously known and novel asymptotics for models such as linear regression, kernel regression, and linear random feature models.
We derive a novel deterministic equivalence for the two-point function of a random matrix resolvent. Using this result, we give a unified derivation of the performance of a wide variety of high-dimensional linear models trained with stochastic gradient descent. This includes high-dimensional linear regression, kernel regression, and linear random feature models. Our results include previously known asymptotics as well as novel ones.