Analysis and Implementation of an Asynchronous Optimization Algorithm for the Parameter Server
This work addresses optimization challenges in distributed computing for machine learning, but it is incremental as it builds on existing asynchronous methods with specific convergence guarantees.
The paper tackles the problem of solving regularized optimization problems in a parameter server framework by presenting an asynchronous incremental aggregated gradient algorithm that handles convex regularizers and constraints, establishing linear convergence rates and explicit step-size choices for strongly convex losses, with simulations validating the results.
This paper presents an asynchronous incremental aggregated gradient algorithm and its implementation in a parameter server framework for solving regularized optimization problems. The algorithm can handle both general convex (possibly non-smooth) regularizers and general convex constraints. When the empirical data loss is strongly convex, we establish linear convergence rate, give explicit expressions for step-size choices that guarantee convergence to the optimum, and bound the associated convergence factors. The expressions have an explicit dependence on the degree of asynchrony and recover classical results under synchronous operation. Simulations and implementations on commercial compute clouds validate our findings.