The Generative Leap: Sharp Sample Complexity for Efficiently Learning Gaussian Multi-Index Models
This work addresses the sample complexity challenge for learning multi-index models, which is incremental as it extends prior results to a more general setting with practical implications for deep neural networks.
The paper tackles the problem of efficiently learning Gaussian multi-index models, where labels depend on inputs through a low-dimensional subspace, and establishes that a sample complexity of Θ(d^{1 ∨ k/2}) is both necessary and sufficient for agnostic estimation of the hidden subspace.
In this work we consider generic Gaussian Multi-index models, in which the labels only depend on the (Gaussian) $d$-dimensional inputs through their projection onto a low-dimensional $r = O_d(1)$ subspace, and we study efficient agnostic estimation procedures for this hidden subspace. We introduce the \emph{generative leap} exponent $k^\star$, a natural extension of the generative exponent from [Damian et al.'24] to the multi-index setting. We first show that a sample complexity of $n=Θ(d^{1 \vee \k/2})$ is necessary in the class of algorithms captured by the Low-Degree-Polynomial framework. We then establish that this sample complexity is also sufficient, by giving an agnostic sequential estimation procedure (that is, requiring no prior knowledge of the multi-index model) based on a spectral U-statistic over appropriate Hermite tensors. We further compute the generative leap exponent for several examples including piecewise linear functions (deep ReLU networks with bias), and general deep neural networks (with $r$-dimensional first hidden layer).