Efficient Symmetric Norm Regression via Linear Sketching
This addresses the problem of scalable regression for machine learning and numerical linear algebra practitioners, offering significant speed and accuracy improvements over prior methods.
The paper tackles efficient algorithms for overconstrained linear regression with symmetric norm loss functions, achieving a (1+ε)-approximate solution in input-sparsity time for Orlicz norms, improving from a d·polylog n-approximation, and providing the first input-sparsity time algorithm with √d·polylog n·mmc(ℓ)-approximation for general symmetric norms.
We provide efficient algorithms for overconstrained linear regression problems with size $n \times d$ when the loss function is a symmetric norm (a norm invariant under sign-flips and coordinate-permutations). An important class of symmetric norms are Orlicz norms, where for a function $G$ and a vector $y \in \mathbb{R}^n$, the corresponding Orlicz norm $\|y\|_G$ is defined as the unique value $α$ such that $\sum_{i=1}^n G(|y_i|/α) = 1$. When the loss function is an Orlicz norm, our algorithm produces a $(1 + \varepsilon)$-approximate solution for an arbitrarily small constant $\varepsilon > 0$ in input-sparsity time, improving over the previously best-known algorithm which produces a $d \cdot \mathrm{polylog} n$-approximate solution. When the loss function is a general symmetric norm, our algorithm produces a $\sqrt{d} \cdot \mathrm{polylog} n \cdot \mathrm{mmc}(\ell)$-approximate solution in input-sparsity time, where $\mathrm{mmc}(\ell)$ is a quantity related to the symmetric norm under consideration. To the best of our knowledge, this is the first input-sparsity time algorithm with provable guarantees for the general class of symmetric norm regression problem. Our results shed light on resolving the universal sketching problem for linear regression, and the techniques might be of independent interest to numerical linear algebra problems more broadly.