Distributed SAGA: Maintaining linear convergence rate with limited communication
This work addresses the problem of efficient distributed optimization for machine learning practitioners using clusters, though it appears incremental as it adapts an existing method to a distributed context.
The paper tackles the challenge of scaling variance-reducing stochastic methods like SAGA to large distributed datasets with limited communication, achieving a linear convergence rate in this setting.
In recent years, variance-reducing stochastic methods have shown great practical performance, exhibiting linear convergence rate when other stochastic methods offered a sub-linear rate. However, as datasets grow ever bigger and clusters become widespread, the need for fast distribution methods is pressing. We propose here a distribution scheme for SAGA which maintains a linear convergence rate, even when communication between nodes is limited.