Tuning the Scheduling of Distributed Stochastic Gradient Descent with Bayesian Optimization
This work addresses the challenge of optimizing distributed SGD scheduling for faster training in machine learning systems, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the problem of tuning system parameters for distributed stochastic gradient descent (SGD) to minimize iteration time, using a Bayesian optimization approach with a custom probabilistic model that exploits runtime measurements. The result shows convergence to efficient configurations within ten iterations, outperforming generic optimizers by up to 2X.
We present an optimizer which uses Bayesian optimization to tune the system parameters of distributed stochastic gradient descent (SGD). Given a specific context, our goal is to quickly find efficient configurations which appropriately balance the load between the available machines to minimize the average SGD iteration time. Our experiments consider setups with over thirty parameters. Traditional Bayesian optimization, which uses a Gaussian process as its model, is not well suited to such high dimensional domains. To reduce convergence time, we exploit the available structure. We design a probabilistic model which simulates the behavior of distributed SGD and use it within Bayesian optimization. Our model can exploit many runtime measurements for inference per evaluation of the objective function. Our experiments show that our resulting optimizer converges to efficient configurations within ten iterations, the optimized configurations outperform those found by generic optimizer in thirty iterations by up to 2X.