Scaling Deep Learning Research with Kubernetes on the NRP Nautilus HyperCluster
This work addresses the problem of accelerating deep learning research for scientists and engineers by scaling training on a specific hypercluster, but it is incremental as it applies existing methods to new data.
The paper tackled the bottleneck of long training times for deep neural networks by using the NRP Nautilus HyperCluster with Kubernetes to automate and scale training across three applications, resulting in the training of 234 models over 4,040 hours.
Throughout the scientific computing space, deep learning algorithms have shown excellent performance in a wide range of applications. As these deep neural networks (DNNs) continue to mature, the necessary compute required to train them has continued to grow. Today, modern DNNs require millions of FLOPs and days to weeks of training to generate a well-trained model. The training times required for DNNs are oftentimes a bottleneck in DNN research for a variety of deep learning applications, and as such, accelerating and scaling DNN training enables more robust and accelerated research. To that end, in this work, we explore utilizing the NRP Nautilus HyperCluster to automate and scale deep learning model training for three separate applications of DNNs, including overhead object detection, burned area segmentation, and deforestation detection. In total, 234 deep neural models are trained on Nautilus, for a total time of 4,040 hours