A Support-Set Algorithm for Optimization Problems with Nonnegative and Orthogonal Constraints
This addresses computational efficiency for constrained optimization problems in machine learning and data analysis, though it appears incremental as it builds on existing methods for specific constraints.
The paper tackles optimization problems with nonnegative and orthogonal constraints by proposing a support-set algorithm that exploits the sparsity structure to compute global solutions in closed form, achieving an iteration complexity of O(ε⁻²) for ε-approximate stationary points and demonstrating strong performance in applications like nonnegative PCA, clustering, and community detection.
In this paper, we investigate optimization problems with nonnegative and orthogonal constraints, where any feasible matrix of size $n \times p$ exhibits a sparsity pattern such that each row accommodates at most one nonzero entry. Our analysis demonstrates that, by fixing the support set, the global solution of the minimization subproblem for the proximal linearization of the objective function can be computed in closed form with at most $n$ nonzero entries. Exploiting this structural property offers a powerful avenue for dramatically enhancing computational efficiency. Guided by this insight, we propose a support-set algorithm preserving strictly the feasibility of iterates. A central ingredient is a strategically devised update scheme for support sets that adjusts the placement of nonzero entries. We establish the global convergence of the support-set algorithm to a first-order stationary point, and show that its iteration complexity required to reach an $ε$-approximate first-order stationary point is $O (ε^{-2})$. Numerical results are strongly in favor of our algorithm in real-world applications, including nonnegative PCA, clustering, and community detection.