Gradient tracking and variance reduction for decentralized optimization and machine learning
This addresses efficient and private decentralized training for machine learning, but it is incremental as it reviews and unifies existing methods.
The paper tackles decentralized optimization for distributed data with privacy constraints by combining gradient tracking with variance reduction, achieving fast convergence with theoretical guarantees for smooth strongly-convex functions and demonstrating applicability to non-convex problems via experiments.
Decentralized methods to solve finite-sum minimization problems are important in many signal processing and machine learning tasks where the data is distributed over a network of nodes and raw data sharing is not permitted due to privacy and/or resource constraints. In this article, we review decentralized stochastic first-order methods and provide a unified algorithmic framework that combines variance-reduction with gradient tracking to achieve both robust performance and fast convergence. We provide explicit theoretical guarantees of the corresponding methods when the objective functions are smooth and strongly-convex, and show their applicability to non-convex problems via numerical experiments. Throughout the article, we provide intuitive illustrations of the main technical ideas by casting appropriate tradeoffs and comparisons among the methods of interest and by highlighting applications to decentralized training of machine learning models.