Subspace Embedding and Linear Regression with Orlicz Norm
This work addresses theoretical optimization problems in machine learning and data analysis, offering incremental algorithmic improvements for specialized norms.
The paper tackles the linear regression problem with Orlicz norm loss, developing an oblivious subspace embedding that maps high-dimensional Orlicz norms to lower-dimensional ℓ₂ norms with distortion bounds, and applies it to approximate regression up to a O(d log² n) factor and improve ℓ_p low-rank matrix approximation for 1 ≤ p < 2.
We consider a generalization of the classic linear regression problem to the case when the loss is an Orlicz norm. An Orlicz norm is parameterized by a non-negative convex function $G:\mathbb{R}_+\rightarrow\mathbb{R}_+$ with $G(0)=0$: the Orlicz norm of a vector $x\in\mathbb{R}^n$ is defined as $ \|x\|_G=\inf\left\{α>0\large\mid\sum_{i=1}^n G(|x_i|/α)\leq 1\right\}. $ We consider the cases where the function $G(\cdot)$ grows subquadratically. Our main result is based on a new oblivious embedding which embeds the column space of a given matrix $A\in\mathbb{R}^{n\times d}$ with Orlicz norm into a lower dimensional space with $\ell_2$ norm. Specifically, we show how to efficiently find an embedding matrix $S\in\mathbb{R}^{m\times n},m<n$ such that $\forall x\in\mathbb{R}^{d},Ω(1/(d\log n)) \cdot \|Ax\|_G\leq \|SAx\|_2\leq O(d^2\log n) \cdot \|Ax\|_G.$ By applying this subspace embedding technique, we show an approximation algorithm for the regression problem $\min_{x\in\mathbb{R}^d} \|Ax-b\|_G$, up to a $O(d\log^2 n)$ factor. As a further application of our techniques, we show how to also use them to improve on the algorithm for the $\ell_p$ low rank matrix approximation problem for $1\leq p<2$.