Ridge Regression and Provable Deterministic Ridge Leverage Score Sampling
This work addresses the need for interpretable and deterministic algorithms in randomized linear algebra and machine learning for practitioners dealing with moderately big data, offering an incremental improvement over existing randomized methods.
The paper tackles the problem of deterministic column sampling for ridge leverage scores, providing provable guarantees for matrix sketching with (1 + ε) error bounds on tasks like column subset selection and projection-cost preservation, and shows it matches randomized algorithms under power-law decay assumptions. It also applies this to ridge regression, offering a method that forces coefficients to zero with a (1 + ε) bound on statistical risk as an alternative to elastic net regularization.
Ridge leverage scores provide a balance between low-rank approximation and regularization, and are ubiquitous in randomized linear algebra and machine learning. Deterministic algorithms are also of interest in the moderately big data regime, because deterministic algorithms provide interpretability to the practitioner by having no failure probability and always returning the same results. We provide provable guarantees for deterministic column sampling using ridge leverage scores. The matrix sketch returned by our algorithm is a column subset of the original matrix, yielding additional interpretability. Like the randomized counterparts, the deterministic algorithm provides (1 + ε) error column subset selection, (1 + ε) error projection-cost preservation, and an additive-multiplicative spectral bound. We also show that under the assumption of power-law decay of ridge leverage scores, this deterministic algorithm is provably as accurate as randomized algorithms. Lastly, ridge regression is frequently used to regularize ill-posed linear least-squares problems. While ridge regression provides shrinkage for the regression coefficients, many of the coefficients remain small but non-zero. Performing ridge regression with the matrix sketch returned by our algorithm and a particular regularization parameter forces coefficients to zero and has a provable (1 + ε) bound on the statistical risk. As such, it is an interesting alternative to elastic net regularization.