Cloud Collectives: Towards Cloud-aware Collectives forML Workloads with Rank Reordering
This addresses performance bottlenecks in distributed ML training in cloud environments, offering a non-intrusive solution without requiring cloud provider support, though it appears incremental as it builds on existing collective algorithms.
The paper tackles the problem of inefficient parameter exchange in cloud-based ML training due to suboptimal collective communication algorithms, and presents Cloud Collectives, a prototype that reorders VM ranks to exploit network locality, achieving speedups of up to 3.7x in microbenchmarks and 1.3x in real-world workloads.
ML workloads are becoming increasingly popular in the cloud. Good cloud training performance is contingent on efficient parameter exchange among VMs. We find that Collectives, the widely used distributed communication algorithms, cannot perform optimally out of the box due to the hierarchical topology of datacenter networks and multi-tenancy nature of the cloudenvironment.In this paper, we present Cloud Collectives , a prototype that accelerates collectives by reordering theranks of participating VMs such that the communication pattern dictated by the selected collectives operation best exploits the locality in the network.Collectives is non-intrusive, requires no code changes nor rebuild of an existing application, and runs without support from cloud providers. Our preliminary application of Cloud Collectives on allreduce operations in public clouds results in a speedup of up to 3.7x in multiple microbenchmarks and 1.3x in real-world workloads of distributed training of deep neural networks and gradient boosted decision trees using state-of-the-art frameworks.