OmniLearn: A Framework for Distributed Deep Learning over Heterogeneous Clusters
This addresses inefficiencies in distributed training for users in edge, cloud, and HPC environments, offering a domain-specific incremental improvement.
The paper tackles the problem of performance degradation in distributed deep learning on heterogeneous clusters due to stragglers and stale updates, developing the OmniLearn framework that reduces training time by 14-85% and improves accuracy by up to 6.9% in asynchronous training.
Deep learning systems are optimized for clusters with homogeneous resources. However, heterogeneity is prevalent in computing infrastructure across edge, cloud and HPC. When training neural networks using stochastic gradient descent techniques on heterogeneous resources, performance degrades due to stragglers and stale updates. In this work, we develop an adaptive batch-scaling framework called OmniLearn to mitigate the effects of heterogeneity in distributed training. Our approach is inspired by proportional controllers to balance computation across heterogeneous servers, and works under varying resource availability. By dynamically adjusting worker mini-batches at runtime, OmniLearn reduces training time by 14-85%. We also investigate asynchronous training, where our techniques improve accuracy by up to 6.9%.