AdaDelay: Delay Adaptive Distributed Stochastic Convex Optimization
This addresses efficiency issues in real-world distributed computation networks where machines have varying speeds, though it appears incremental as it builds on existing delayed gradient models.
The paper tackles distributed stochastic convex optimization with heterogeneous worker delays by making parameter updates sensitive to actual delays rather than worst-case bounds, enabling larger stepsizes for faster initial convergence while maintaining asymptotic complexity. Experiments on real datasets with billions of examples and features show encouraging improvements in overall convergence.
We study distributed stochastic convex optimization under the delayed gradient model where the server nodes perform parameter updates, while the worker nodes compute stochastic gradients. We discuss, analyze, and experiment with a setup motivated by the behavior of real-world distributed computation networks, where the machines are differently slow at different time. Therefore, we allow the parameter updates to be sensitive to the actual delays experienced, rather than to worst-case bounds on the maximum delay. This sensitivity leads to larger stepsizes, that can help gain rapid initial convergence without having to wait too long for slower machines, while maintaining the same asymptotic complexity. We obtain encouraging improvements to overall convergence for distributed experiments on real datasets with up to billions of examples and features.