Training Time Prediction for Mixed Precision-based Distributed Training
For cloud operators and researchers, this work improves training time prediction accuracy by accounting for precision, a previously overlooked factor.
Existing distributed training time predictors ignore floating-point precision, causing up to 147.85% MAPE. A precision-aware predictor achieves 9.8% MAPE across diverse precision settings including mixed precision.
Accurate prediction of training time in distributed deep learning is crucial for resource allocation, cost estimation, and job scheduling. We observe that the floating-point precision setting is a key determinant of training time, leading to training time variations of ~2.4x over its minimum. However, existing studies on distributed training time prediction rely on static model computation graphs that do not capture precision variations, including mixed precision. According to our experiments, training time prediction without considering precision results in significant prediction errors - reaching up to 147.85% in mean absolute percentage error (MAPE). To address this issue, we propose a precision-aware distributed training time predictor that achieves robust accuracy across diverse precision settings, including mixed precision, with 9.8% MAPE.