Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching
This addresses inefficiencies in distributed training for users of low-cost, transient resources like EC2 spot instances, though it is incremental as it builds on existing data-parallel methods.
The paper tackles the problem of distributed machine learning training on heterogeneous clusters by developing a dynamic batching technique that adjusts mini-batch sizes per worker based on resource availability, reducing training times by more than 4x in empirical evaluations.
Current techniques and systems for distributed model training mostly assume that clusters are comprised of homogeneous servers with a constant resource availability. However, cluster heterogeneity is pervasive in computing infrastructure, and is a fundamental characteristic of low-cost transient resources (such as EC2 spot instances). In this paper, we develop a dynamic batching technique for distributed data-parallel training that adjusts the mini-batch sizes on each worker based on its resource availability and throughput. Our mini-batch controller seeks to equalize iteration times on all workers, and facilitates training on clusters comprised of servers with different amounts of CPU and GPU resources. This variable mini-batch technique uses proportional control and ideas from PID controllers to find stable mini-batch sizes. Our empirical evaluation shows that dynamic batching can reduce model training times by more than 4x on heterogeneous clusters.