The Squared-Error of Generalized LASSO: A Precise Analysis
This work provides theoretical guarantees for LASSO-based estimation in compressed sensing and statistics, offering precise error bounds that can guide parameter selection and algorithm design, though it is incremental in extending prior analyses.
The paper tackles the problem of precisely analyzing the normalized squared error of generalized LASSO estimators for structured signal recovery from noisy linear observations, deriving exact error formulas that depend on Gaussian distance parameters and providing closed-form upper bounds for important signal classes.
We consider the problem of estimating an unknown signal $x_0$ from noisy linear observations $y = Ax_0 + z\in R^m$. In many practical instances, $x_0$ has a certain structure that can be captured by a structure inducing convex function $f(\cdot)$. For example, $\ell_1$ norm can be used to encourage a sparse solution. To estimate $x_0$ with the aid of $f(\cdot)$, we consider the well-known LASSO method and provide sharp characterization of its performance. We assume the entries of the measurement matrix $A$ and the noise vector $z$ have zero-mean normal distributions with variances $1$ and $σ^2$ respectively. For the LASSO estimator $x^*$, we attempt to calculate the Normalized Square Error (NSE) defined as $\frac{\|x^*-x_0\|_2^2}{σ^2}$ as a function of the noise level $σ$, the number of observations $m$ and the structure of the signal. We show that, the structure of the signal $x_0$ and choice of the function $f(\cdot)$ enter the error formulae through the summary parameters $D(cone)$ and $D(λ)$, which are defined as the Gaussian squared-distances to the subdifferential cone and to the $λ$-scaled subdifferential, respectively. The first LASSO estimator assumes a-priori knowledge of $f(x_0)$ and is given by $\arg\min_{x}\{{\|y-Ax\|_2}~\text{subject to}~f(x)\leq f(x_0)\}$. We prove that its worst case NSE is achieved when $σ\rightarrow 0$ and concentrates around $\frac{D(cone)}{m-D(cone)}$. Secondly, we consider $\arg\min_{x}\{\|y-Ax\|_2+λf(x)\}$, for some $λ\geq 0$. This time the NSE formula depends on the choice of $λ$ and is given by $\frac{D(λ)}{m-D(λ)}$. We then establish a mapping between this and the third estimator $\arg\min_{x}\{\frac{1}{2}\|y-Ax\|_2^2+ λf(x)\}$. Finally, for a number of important structured signal classes, we translate our abstract formulae to closed-form upper bounds on the NSE.