Compute, Time and Energy Characterization of Encoder-Decoder Networks with Automatic Mixed Precision Training
This addresses efficiency challenges for researchers and practitioners training large models in domains like weather forecasting, though it is incremental as it applies existing optimizations to a specific problem.
The paper tackled the high compute, time, and energy costs of training deep neural networks for weather forecasting by applying data-distributed and mixed-precision training to a UNet model, achieving significant training time improvements without performance loss and showing that a 1549% increase in parameters led to only a 63.22% increase in energy usage.
Deep neural networks have shown great success in many diverse fields. The training of these networks can take significant amounts of time, compute and energy. As datasets get larger and models become more complex, the exploration of model architectures becomes prohibitive. In this paper we examine the compute, energy and time costs of training a UNet based deep neural network for the problem of predicting short term weather forecasts (called precipitation Nowcasting). By leveraging a combination of data distributed and mixed-precision training, we explore the design space for this problem. We also show that larger models with better performance come at a potentially incremental cost if appropriate optimizations are used. We show that it is possible to achieve a significant improvement in training time by leveraging mixed-precision training without sacrificing model performance. Additionally, we find that a 1549% increase in the number of trainable parameters for a network comes at a relatively smaller 63.22% increase in energy usage for a UNet with 4 encoding layers.