Model Selection for High-Dimensional Regression under the Generalized Irrepresentability Condition
This work addresses model selection for statisticians and data scientists dealing with high-dimensional data, offering a theoretical improvement over existing Lasso-based methods by relaxing assumptions.
The paper tackles the problem of model selection in high-dimensional regression where the sample size is smaller than the number of covariates, by proposing the Gauss-Lasso selector, a two-stage method that first uses Lasso and then ordinary least squares. Under a weaker generalized irrepresentability condition, it proves that this method correctly recovers the active set of covariates.
In the high-dimensional regression model a response variable is linearly related to $p$ covariates, but the sample size $n$ is smaller than $p$. We assume that only a small subset of covariates is `active' (i.e., the corresponding coefficients are non-zero), and consider the model-selection problem of identifying the active covariates. A popular approach is to estimate the regression coefficients through the Lasso ($\ell_1$-regularized least squares). This is known to correctly identify the active set only if the irrelevant covariates are roughly orthogonal to the relevant ones, as quantified through the so called `irrepresentability' condition. In this paper we study the `Gauss-Lasso' selector, a simple two-stage method that first solves the Lasso, and then performs ordinary least squares restricted to the Lasso active set. We formulate `generalized irrepresentability condition' (GIC), an assumption that is substantially weaker than irrepresentability. We prove that, under GIC, the Gauss-Lasso correctly recovers the active set.