Asymptotic Network Independence in Distributed Stochastic Optimization for Machine Learning
This addresses the problem of efficient distributed training for machine learning models, but it is incremental as it reviews and explains existing results rather than introducing new methods.
The paper discusses recent results that overcome a barrier in distributed stochastic optimization for machine learning by achieving asymptotic network independence, where distributed methods converge to optimal solutions at rates comparable to centralized methods with equivalent computational power.
We provide a discussion of several recent results which, in certain scenarios, are able to overcome a barrier in distributed stochastic optimization for machine learning. Our focus is the so-called asymptotic network independence property, which is achieved whenever a distributed method executed over a network of n nodes asymptotically converges to the optimal solution at a comparable rate to a centralized method with the same computational power as the entire network. We explain this property through an example involving the training of ML models and sketch a short mathematical analysis for comparing the performance of distributed stochastic gradient descent (DSGD) with centralized stochastic gradient decent (SGD).