Learning a Single Index Model from Anisotropic Data with vanilla Stochastic Gradient Descent
This work addresses a gap in understanding learning dynamics for anisotropic data in neural network feature learning, though it appears incremental by extending isotropic case analyses.
The authors tackled the problem of learning a Single Index Model from anisotropic Gaussian data using vanilla Stochastic Gradient Descent, showing that it automatically adapts to the covariance structure and deriving sample complexity bounds based on an effective dimension rather than input dimension.
We investigate the problem of learning a Single Index Model (SIM)- a popular model for studying the ability of neural networks to learn features - from anisotropic Gaussian inputs by training a neuron using vanilla Stochastic Gradient Descent (SGD). While the isotropic case has been extensively studied, the anisotropic case has received less attention and the impact of the covariance matrix on the learning dynamics remains unclear. For instance, Mousavi-Hosseini et al. (2023b) proposed a spherical SGD that requires a separate estimation of the data covariance matrix, thereby oversimplifying the influence of covariance. In this study, we analyze the learning dynamics of vanilla SGD under the SIM with anisotropic input data, demonstrating that vanilla SGD automatically adapts to the data's covariance structure. Leveraging these results, we derive upper and lower bounds on the sample complexity using a notion of effective dimension that is determined by the structure of the covariance matrix instead of the input data dimension.