Ridge Regression Revisited: Debiasing, Thresholding and Bootstrap
This work addresses the problem of improving ridge regression for high-dimensional data analysis, offering a method that may be preferable in some settings, though it is incremental as it builds on existing debiasing and thresholding techniques.
The paper tackles the limitations of ridge regression in high-dimensional settings by proposing a debiased and thresholded version, proving consistency and Gaussian approximation results, and introducing bootstrap methods for confidence regions and prediction intervals, with simulations showing favorable finite-sample performance.
The success of the Lasso in the era of high-dimensional data can be attributed to its conducting an implicit model selection, i.e., zeroing out regression coefficients that are not significant. By contrast, classical ridge regression can not reveal a potential sparsity of parameters, and may also introduce a large bias under the high-dimensional setting. Nevertheless, recent work on the Lasso involves debiasing and thresholding, the latter in order to further enhance the model selection. As a consequence, ridge regression may be worth another look since -- after debiasing and thresholding -- it may offer some advantages over the Lasso, e.g., it can be easily computed using a closed-form expression. % and it has similar performance to threshold Lasso. In this paper, we define a debiased and thresholded ridge regression method, and prove a consistency result and a Gaussian approximation theorem. We further introduce a wild bootstrap algorithm to construct confidence regions and perform hypothesis testing for a linear combination of parameters. In addition to estimation, we consider the problem of prediction, and present a novel, hybrid bootstrap algorithm tailored for prediction intervals. Extensive numerical simulations further show that the debiased and thresholded ridge regression has favorable finite sample performance and may be preferable in some settings.