Understand Data Preprocessing for Effective End-to-End Training of Deep Neural Networks
This addresses performance inefficiencies for ML practitioners using cloud-based DNN training, though it is incremental in optimizing existing methods.
The paper investigates data preprocessing as a bottleneck in deep neural network training on public clouds, finding that even with optimized libraries like NVIDIA DALI, preprocessing significantly impacts performance. It analyzes causes and optimization methods to guide co-design of data pipelines and training frameworks for better resource utilization.
In this paper, we primarily focus on understanding the data preprocessing pipeline for DNN Training in the public cloud. First, we run experiments to test the performance implications of the two major data preprocessing methods using either raw data or record files. The preliminary results show that data preprocessing is a clear bottleneck, even with the most efficient software and hardware configuration enabled by NVIDIA DALI, a high-optimized data preprocessing library. Second, we identify the potential causes, exercise a variety of optimization methods, and present their pros and cons. We hope this work will shed light on the new co-design of ``data storage, loading pipeline'' and ``training framework'' and flexible resource configurations between them so that the resources can be fully exploited and performance can be maximized.