Finite-sample and asymptotic analysis of generalization ability with an application to penalized regression
This work addresses the theoretical underpinnings of model generalization for statisticians and machine learning practitioners, offering incremental improvements in error analysis and hyperparameter selection.
The paper tackles the problem of understanding generalization ability in extremum estimators, deriving finite-sample and asymptotic error bounds for penalized regression and showing how these bounds can guide hyperparameter tuning and explain differences between penalized and unpenalized estimates, with proven L2-consistency across both n ≥ p and n < p cases.
In this paper, we study the performance of extremum estimators from the perspective of generalization ability (GA): the ability of a model to predict outcomes in new samples from the same population. By adapting the classical concentration inequalities, we derive upper bounds on the empirical out-of-sample prediction errors as a function of the in-sample errors, in-sample data size, heaviness in the tails of the error distribution, and model complexity. We show that the error bounds may be used for tuning key estimation hyper-parameters, such as the number of folds $K$ in cross-validation. We also show how $K$ affects the bias-variance trade-off for cross-validation. We demonstrate that the $\mathcal{L}_2$-norm difference between penalized and the corresponding un-penalized regression estimates is directly explained by the GA of the estimates and the GA of empirical moment conditions. Lastly, we prove that all penalized regression estimates are $L_2$-consistent for both the $n \geqslant p$ and the $n < p$ cases. Simulations are used to demonstrate key results. Keywords: generalization ability, upper bound of generalization error, penalized regression, cross-validation, bias-variance trade-off, $\mathcal{L}_2$ difference between penalized and unpenalized regression, lasso, high-dimensional data.