How does feature learning reshape the function space?
For theorists and practitioners studying neural network generalization, this work offers a rigorous understanding of how feature learning modifies the function space, distinguishing it from fixed-kernel methods.
The paper characterizes how feature learning in two-layer neural networks reshapes the function space during gradient descent, showing that it induces a data-adaptive kernel that selectively amplifies eigenvalues aligned with the target direction. The analysis provides a precise function-space perspective on early-stage feature learning.
Feature learning is widely regarded as the key mechanism distinguishing neural networks from fixed-kernel methods, yet its impact on the induced function space remains poorly understood. In this work, we precisely characterize how the function space spanned by the features of a two-layer neural network evolves during gradient descent training. We prove that, in the high-dimensional proportional regime, after a large gradient step the post-update feature distribution is well approximated by a target-dependent spiked Gaussian covariance. This induces a data-adaptive kernel that reshapes the function space and modifies its spectral structure. Our analysis reveals that feature learning can be interpreted as a distributional transformation in either parameter space or input space, equivalently as the introduction of a target-dependent kernel. In particular, it selectively amplifies eigenvalues aligned with the target direction and mixes leading eigenfunctions, coupling the top radial mode with a target-aligned quadratic harmonic. Overall, our results provide a precise function-space perspective on early-stage feature learning: rather than just rescaling a fixed kernel, gradient descent induces a data-adaptive deformation that preferentially enhances directions aligned with the signal in the data.