ASAGA: Asynchronous Parallel SAGA
This addresses the problem of efficient parallel optimization for machine learning practitioners, offering a significant but incremental improvement over existing methods.
The paper tackles the problem of slow convergence in asynchronous parallel optimization by proposing ASAGA, an asynchronous parallel version of SAGA that achieves fast linear convergence rates. The result shows ASAGA can obtain a theoretical linear speedup on multi-core systems without sparsity assumptions, with implementation on a 40-core architecture illustrating practical speedup and hardware overhead.
We describe ASAGA, an asynchronous parallel version of the incremental gradient algorithm SAGA that enjoys fast linear convergence rates. Through a novel perspective, we revisit and clarify a subtle but important technical issue present in a large fraction of the recent convergence rate proofs for asynchronous parallel optimization algorithms, and propose a simplification of the recently introduced "perturbed iterate" framework that resolves it. We thereby prove that ASAGA can obtain a theoretical linear speedup on multi-core systems even without sparsity assumptions. We present results of an implementation on a 40-core architecture illustrating the practical speedup as well as the hardware overhead.