Gradient flow in the gaussian covariate model: exact solution of learning curves and multiple descent structures
This work addresses the theoretical understanding of multiple descent phenomena in machine learning, offering a foundational analysis that unifies and extends prior observations, though it is incremental in building on existing random matrix theory methods.
The authors tackled the problem of understanding the full time-evolution of generalization error curves in learning models, providing a unified analytical solution in the asymptotic large-dimensional regime under gradient-flow for a Gaussian covariate model, which matches learning curves from realistic datasets and reveals multiple descent structures as functions of model parameters or time.
A recent line of work has shown remarkable behaviors of the generalization error curves in simple learning models. Even the least-squares regression has shown atypical features such as the model-wise double descent, and further works have observed triple or multiple descents. Another important characteristic are the epoch-wise descent structures which emerge during training. The observations of model-wise and epoch-wise descents have been analytically derived in limited theoretical settings (such as the random feature model) and are otherwise experimental. In this work, we provide a full and unified analysis of the whole time-evolution of the generalization curve, in the asymptotic large-dimensional regime and under gradient-flow, within a wider theoretical setting stemming from a gaussian covariate model. In particular, we cover most cases already disparately observed in the literature, and also provide examples of the existence of multiple descent structures as a function of a model parameter or time. Furthermore, we show that our theoretical predictions adequately match the learning curves obtained by gradient descent over realistic datasets. Technically we compute averages of rational expressions involving random matrices using recent developments in random matrix theory based on "linear pencils". Another contribution, which is also of independent interest in random matrix theory, is a new derivation of related fixed point equations (and an extension there-off) using Dyson brownian motions.