On the Suboptimality of Proximal Gradient Descent for $\ell^{0}$ Sparse Approximation
This work addresses computational efficiency in sparse approximation for signal processing or machine learning, but it is incremental as it builds on existing PGD methods with randomized accelerations.
The paper tackles the problem of solving the ℓ⁰ sparse approximation problem using proximal gradient descent (PGD), showing that PGD yields a solution with a bounded gap to the global optimum under weaker conditions than the Restricted Isometry Property. It also proposes randomized algorithms (PGD-RMA and PGD-RDR) that reduce computation costs while maintaining provable suboptimality bounds.
We study the proximal gradient descent (PGD) method for $\ell^{0}$ sparse approximation problem as well as its accelerated optimization with randomized algorithms in this paper. We first offer theoretical analysis of PGD showing the bounded gap between the sub-optimal solution by PGD and the globally optimal solution for the $\ell^{0}$ sparse approximation problem under conditions weaker than Restricted Isometry Property widely used in compressive sensing literature. Moreover, we propose randomized algorithms to accelerate the optimization by PGD using randomized low rank matrix approximation (PGD-RMA) and randomized dimension reduction (PGD-RDR). Our randomized algorithms substantially reduces the computation cost of the original PGD for the $\ell^{0}$ sparse approximation problem, and the resultant sub-optimal solution still enjoys provable suboptimality, namely, the sub-optimal solution to the reduced problem still has bounded gap to the globally optimal solution to the original problem.