On Single Index Models beyond Gaussian Data
This work addresses the challenge of feature learning in high-dimensional data for machine learning practitioners, but it is incremental as it builds on prior frameworks.
The paper tackles the problem of efficiently recovering the unknown direction in single-index models beyond Gaussian data, showing that Stochastic Gradient Descent can achieve this under extended assumptions.
Sparse high-dimensional functions have arisen as a rich framework to study the behavior of gradient-descent methods using shallow neural networks, showcasing their ability to perform feature learning beyond linear models. Amongst those functions, the simplest are single-index models $f(x) = φ( x \cdot θ^*)$, where the labels are generated by an arbitrary non-linear scalar link function $φ$ applied to an unknown one-dimensional projection $θ^*$ of the input data. By focusing on Gaussian data, several recent works have built a remarkable picture, where the so-called information exponent (related to the regularity of the link function) controls the required sample complexity. In essence, these tools exploit the stability and spherical symmetry of Gaussian distributions. In this work, building from the framework of \cite{arous2020online}, we explore extensions of this picture beyond the Gaussian setting, where both stability or symmetry might be violated. Focusing on the planted setting where $φ$ is known, our main results establish that Stochastic Gradient Descent can efficiently recover the unknown direction $θ^*$ in the high-dimensional regime, under assumptions that extend previous works \cite{yehudai2020learning,wu2022learning}.