Importance of Data Loading Pipeline in Training Deep Neural Networks
This work addresses training efficiency for practitioners using large-scale deep learning models, but it is incremental as it compares existing tools rather than introducing new methods.
The paper tackled the problem of slow training times in deep neural networks due to inefficient data loading and augmentation pipelines, showing that using dedicated tools like binary data format and NVIDIA DALI can improve training speed by 20% to 40%.
Training large-scale deep neural networks is a long, time-consuming operation, often requiring many GPUs to accelerate. In large models, the time spent loading data takes a significant portion of model training time. As GPU servers are typically expensive, tricks that can save training time are valuable.Slow training is observed especially on real-world applications where exhaustive data augmentation operations are required. Data augmentation techniques include: padding, rotation, adding noise, down sampling, up sampling, etc. These additional operations increase the need to build an efficient data loading pipeline, and to explore existing tools to speed up training time. We focus on the comparison of two main tools designed for this task, namely binary data format to accelerate data reading, and NVIDIA DALI to accelerate data augmentation. Our study shows improvement on the order of 20% to 40% if such dedicated tools are used.