Singular ridge regression with homoscedastic residuals: generalization error with estimated parameters
This work addresses theoretical gaps in ridge regression error analysis for statisticians and machine learning researchers, but it is incremental as it builds on classical references with specific assumptions.
The paper tackles the problem of characterizing generalization error in ridge regression when parameters are estimated, providing explicit formulas for total regression and generalization errors in a singular setup without assuming a solution exists for the non-regularized problem, and it introduces a conditional homoscedasticity hypothesis on residuals to achieve this.
This paper characterizes the conditional distribution properties of the finite sample ridge regression estimator and uses that result to evaluate total regression and generalization errors that incorporate the inaccuracies committed at the time of parameter estimation. The paper provides explicit formulas for those errors. Unlike other classical references in this setup, our results take place in a fully singular setup that does not assume the existence of a solution for the non-regularized regression problem. In exchange, we invoke a conditional homoscedasticity hypothesis on the regularized regression residuals that is crucial in our developments.