Characterizing and Modeling Distributed Training with Transient Cloud GPU Servers
This work addresses the problem of cost-effective and efficient distributed training for deep learning practitioners using cloud GPU servers, but it is incremental as it builds on existing performance modeling approaches.
The paper tackled the challenge of selecting optimal cloud GPU cluster configurations for distributed deep learning training, balancing training time, cost, and accuracy, and demonstrated the feasibility of predicting training speed and overhead using regression-based models based on empirical measurements from thousands of servers.
Cloud GPU servers have become the de facto way for deep learning practitioners to train complex models on large-scale datasets. However, it is challenging to determine the appropriate cluster configuration---e.g., server type and number---for different training workloads while balancing the trade-offs in training time, cost, and model accuracy. Adding to the complexity is the potential to reduce the monetary cost by using cheaper, but revocable, transient GPU servers. In this work, we analyze distributed training performance under diverse cluster configurations using CM-DARE, a cloud-based measurement and training framework. Our empirical datasets include measurements from three GPU types, six geographic regions, twenty convolutional neural networks, and thousands of Google Cloud servers. We also demonstrate the feasibility of predicting training speed and overhead using regression-based models. Finally, we discuss potential use cases of our performance modeling such as detecting and mitigating performance bottlenecks.