Decentralized Weakly Convex Optimization Over the Stiefel Manifold
This addresses a challenging problem in decentralized machine learning for applications involving non-smooth, non-convex optimization, representing an incremental advancement with specific theoretical guarantees.
The paper tackles decentralized non-smooth optimization over the Stiefel manifold by proposing the decentralized Riemannian subgradient method (DRSM), achieving an iteration complexity of O(ε^{-2} log^2(ε^{-1})) for stationarity and local linear convergence under sharpness conditions.
We focus on a class of non-smooth optimization problems over the Stiefel manifold in the decentralized setting, where a connected network of $n$ agents cooperatively minimize a finite-sum objective function with each component being weakly convex in the ambient Euclidean space. Such optimization problems, albeit frequently encountered in applications, are quite challenging due to their non-smoothness and non-convexity. To tackle them, we propose an iterative method called the decentralized Riemannian subgradient method (DRSM). The global convergence and an iteration complexity of $\mathcal{O}(\varepsilon^{-2} \log^2(\varepsilon^{-1}))$ for forcing a natural stationarity measure below $\varepsilon$ are established via the powerful tool of proximal smoothness from variational analysis, which could be of independent interest. Besides, we show the local linear convergence of the DRSM using geometrically diminishing stepsizes when the problem at hand further possesses a sharpness property. Numerical experiments are conducted to corroborate our theoretical findings.