Learning Under Delayed Feedback: Implicitly Adapting to Gradient Delays
This addresses the challenge of efficient machine learning in shared-resource computational environments where delays are unpredictable, offering a more practical solution for distributed optimization.
The paper tackles the problem of stochastic convex optimization under asynchronous parallel execution with unknown and varying gradient delays, proposing a robust training method that does not require prior knowledge of delays, smoothness, or variance, unlike existing methods that fail in dynamic environments like clouds and data centers.
We consider stochastic convex optimization problems, where several machines act asynchronously in parallel while sharing a common memory. We propose a robust training method for the constrained setting and derive non asymptotic convergence guarantees that do not depend on prior knowledge of update delays, objective smoothness, and gradient variance. Conversely, existing methods for this setting crucially rely on this prior knowledge, which render them unsuitable for essentially all shared-resources computational environments, such as clouds and data centers. Concretely, existing approaches are unable to accommodate changes in the delays which result from dynamic allocation of the machines, while our method implicitly adapts to such changes.