Penalized versus constrained generalized eigenvalue problems
This addresses a specific bottleneck in statistical learning for researchers and practitioners using sparse methods, offering an incremental improvement by highlighting the constraint's superiority over the penalty.
The paper tackles the problem of variable selection in generalized eigenvalue problems like PCA and discriminant analysis, finding that using an ℓ₁ penalty can fail to produce sparse solutions, whereas an ℓ₁ constraint effectively remedies this issue, as supported by empirical and theoretical evidence.
We investigate the difference between using an $\ell_1$ penalty versus an $\ell_1$ constraint in generalized eigenvalue problems, such as principal component analysis and discriminant analysis. Our main finding is that an $\ell_1$ penalty may fail to provide very sparse solutions; a severe disadvantage for variable selection that can be remedied by using an $\ell_1$ constraint. Our claims are supported both by empirical evidence and theoretical analysis. Finally, we illustrate the advantages of an $\ell_1$ constraint in the context of discriminant analysis and principal component analysis.