No penalty no tears: Least squares in high-dimensional linear models
This provides a computationally simple alternative to penalized regression for high-dimensional linear modeling problems in statistics and data science.
The authors tackled the problem of fitting linear models when dimensionality exceeds sample size, where ordinary least squares fails, by proposing two novel three-step algorithms combining generalized least squares with hard thresholding. Their methods demonstrated great potential in simulations and data analyses compared to penalization-based approaches.
Ordinary least squares (OLS) is the default method for fitting linear models, but is not applicable for problems with dimensionality larger than the sample size. For these problems, we advocate the use of a generalized version of OLS motivated by ridge regression, and propose two novel three-step algorithms involving least squares fitting and hard thresholding. The algorithms are methodologically simple to understand intuitively, computationally easy to implement efficiently, and theoretically appealing for choosing models consistently. Numerical exercises comparing our methods with penalization-based approaches in simulations and data analyses illustrate the great potential of the proposed algorithms.