A Variance-Reduced Stochastic Gradient Tracking Algorithm for Decentralized Optimization with Orthogonality Constraints
This addresses a challenging problem in scientific computing and data science, offering a more efficient solution for applications like autonomous driving, though it appears incremental as it builds on existing geometric tools.
The paper tackles decentralized optimization with nonconvex orthogonality constraints, proposing a variance-reduced stochastic gradient tracking algorithm that achieves an O(1/k) convergence rate to a stationary point and reduces both sampling and communication complexities.
Decentralized optimization with orthogonality constraints is found widely in scientific computing and data science. Since the orthogonality constraints are nonconvex, it is quite challenging to design efficient algorithms. Existing approaches leverage the geometric tools from Riemannian optimization to solve this problem at the cost of high sample and communication complexities. To relieve this difficulty, based on two novel techniques that can waive the orthogonality constraints, we propose a variance-reduced stochastic gradient tracking (VRSGT) algorithm with the convergence rate of $O(1 / k)$ to a stationary point. To the best of our knowledge, VRSGT is the first algorithm for decentralized optimization with orthogonality constraints that reduces both sampling and communication complexities simultaneously. In the numerical experiments, VRSGT has a promising performance in a real-world autonomous driving application.