Incorporating priors in learning: a random matrix study under a teacher-student framework
This work addresses a foundational gap in machine learning theory for researchers and practitioners, offering conceptual clarity and practical guidance for learning with structured prior knowledge, though it is incremental in extending existing frameworks.
The authors tackled the problem of understanding high-dimensional behavior in regularized linear regression with informative priors, providing the first exact asymptotic characterization of training and test risks for MAP regression with Gaussian priors, which yields closed-form formulas explaining bias-variance tradeoffs and double descent.
Regularized linear regression is central to machine learning, yet its high-dimensional behavior with informative priors remains poorly understood. We provide the first exact asymptotic characterization of training and test risks for maximum a posteriori (MAP) regression with Gaussian priors centered at a domain-informed initialization. Our framework unifies ridge regression, least squares, and prior-informed estimators, and -- using random matrix theory -- yields closed-form risk formulas that expose the bias-variance-prior tradeoff, explain double descent, and quantify prior mismatch. We also identify a closed-form minimizer of test risk, enabling a simple estimator of the optimal regularization parameter. Simulations confirm the theory with high accuracy. By connecting Bayesian priors, classical regularization, and modern asymptotics, our results provide both conceptual clarity and practical guidance for learning with structured prior knowledge.