Does SLOPE outperform bridge regression?
This provides theoretical insights into the performance of SLOPE estimators for high-dimensional linear regression, but it is incremental as it focuses on a specific regime and compares to existing methods.
The paper analyzes the estimation error of the SLOPE estimator in a regime where sparsity and sample size scale linearly with dimension, showing that LASSO is optimal among SLOPE estimators for sparse vectors without tied components in low noise, but SLOPE is suboptimal compared to bridge regression like Ridge in high noise.
A recently proposed SLOPE estimator (arXiv:1407.3824) has been shown to adaptively achieve the minimax $\ell_2$ estimation rate under high-dimensional sparse linear regression models (arXiv:1503.08393). Such minimax optimality holds in the regime where the sparsity level $k$, sample size $n$, and dimension $p$ satisfy $k/p \rightarrow 0$, $k\log p/n \rightarrow 0$. In this paper, we characterize the estimation error of SLOPE under the complementary regime where both $k$ and $n$ scale linearly with $p$, and provide new insights into the performance of SLOPE estimators. We first derive a concentration inequality for the finite sample mean square error (MSE) of SLOPE. The quantity that MSE concentrates around takes a complicated and implicit form. With delicate analysis of the quantity, we prove that among all SLOPE estimators, LASSO is optimal for estimating $k$-sparse parameter vectors that do not have tied non-zero components in the low noise scenario. On the other hand, in the large noise scenario, the family of SLOPE estimators are sub-optimal compared with bridge regression such as the Ridge estimator.