Fast Escape, Slow Convergence: Learning Dynamics of Phase Retrieval under Power-Law Data
This provides the first rigorous characterization of scaling laws in nonlinear regression with anisotropic data, addressing a foundational issue in machine learning theory for researchers in deep learning dynamics.
The paper tackled the problem of understanding learning dynamics in phase retrieval with anisotropic Gaussian inputs, deriving explicit scaling laws for mean-squared error that show how spectral decay dictates convergence times and error curves, with experiments confirming the predicted phases and exponents.
Scaling laws describe how learning performance improves with data, compute, or training time, and have become a central theme in modern deep learning. We study this phenomenon in a canonical nonlinear model: phase retrieval with anisotropic Gaussian inputs whose covariance spectrum follows a power law. Unlike the isotropic case, where dynamics collapse to a two-dimensional system, anisotropy yields a qualitatively new regime in which an infinite hierarchy of coupled equations governs the evolution of the summary statistics. We develop a tractable reduction that reveals a three-phase trajectory: (i) fast escape from low alignment, (ii) slow convergence of the summary statistics, and (iii) spectral-tail learning in low-variance directions. From this decomposition, we derive explicit scaling laws for the mean-squared error, showing how spectral decay dictates convergence times and error curves. Experiments confirm the predicted phases and exponents. These results provide the first rigorous characterization of scaling laws in nonlinear regression with anisotropic data, highlighting how anisotropy reshapes learning dynamics.