Elastic Bulk Synchronous Parallel Model for Distributed Deep Learning
This addresses synchronization inefficiencies in distributed training for machine learning practitioners, offering an incremental improvement over existing models.
The paper tackles the straggler problem in the bulk synchronous parallel (BSP) model for distributed deep learning by proposing ELASTICBSP, which relaxes synchronization requirements, resulting in faster convergence and higher accuracy than classic BSP, with comparable or better accuracy than other models.
The bulk synchronous parallel (BSP) is a celebrated synchronization model for general-purpose parallel computing that has successfully been employed for distributed training of machine learning models. A prevalent shortcoming of the BSP is that it requires workers to wait for the straggler at every iteration. To ameliorate this shortcoming of classic BSP, we propose ELASTICBSP a model that aims to relax its strict synchronization requirement. The proposed model offers more flexibility and adaptability during the training phase, without sacrificing on the accuracy of the trained model. We also propose an efficient method that materializes the model, named ZIPLINE. The algorithm is tunable and can effectively balance the trade-off between quality of convergence and iteration throughput, in order to accommodate different environments or applications. A thorough experimental evaluation demonstrates that our proposed ELASTICBSP model converges faster and to a higher accuracy than the classic BSP. It also achieves comparable (if not higher) accuracy than the other sensible synchronization models.