Precise analysis of ridge interpolators under heavy correlations -- a Random Duality Theory view
This work offers incremental theoretical insights for researchers in statistical learning by extending risk analysis to heavily correlated settings.
The paper tackles the analysis of classical estimators in fully correlated linear regression models, showing that Random Duality Theory provides precise closed-form characterizations of prediction risk, including non-monotonic double-descent behavior, with results matching prior spectral methods in uncorrelated cases.
We consider fully row/column-correlated linear regression models and study several classical estimators (including minimum norm interpolators (GLS), ordinary least squares (LS), and ridge regressors). We show that \emph{Random Duality Theory} (RDT) can be utilized to obtain precise closed form characterizations of all estimators related optimizing quantities of interest, including the \emph{prediction risk} (testing or generalization error). On a qualitative level out results recover the risk's well known non-monotonic (so-called double-descent) behavior as the number of features/sample size ratio increases. On a quantitative level, our closed form results show how the risk explicitly depends on all key model parameters, including the problem dimensions and covariance matrices. Moreover, a special case of our results, obtained when intra-sample (or time-series) correlations are not present, precisely match the corresponding ones obtained via spectral methods in [6,16,17,24].