Efficient Embedding of MPI Collectives in MXNET DAGs for scaling Deep Learning
This addresses performance bottlenecks in distributed deep learning for researchers and engineers using high-performance computing clusters, though it is incremental as it builds on existing frameworks like MXNET.
The paper tackles the problem of efficiently embedding MPI collective operations into data-parallel DAGs in deep learning frameworks to avoid deadlocks and crashes, achieving scaling to 256 GPUs with epoch times as low as 50 seconds for ImageNet 1K datasets.
Availability of high performance computing infrastructures such as clusters of GPUs and CPUs have fueled the growth of distributed learning systems. Deep Learning frameworks express neural nets as DAGs and execute these DAGs on computation resources such as GPUs. In this paper, we propose efficient designs of embedding MPI collective operations into data parallel DAGs. Incorrect designs can easily lead to deadlocks or program crashes. In particular, we demonstrate three designs: Funneled, Concurrent communication and Dependency chaining of using MPI collectives with DAGs. These designs automatically enable overlap of computation with communication by allowing for concurrent execution with the other tasks. We directly implement these designs into the KVStore API of the MXNET. This allows us to directly leverage the rest of the infrastructure. Using ImageNet and CIFAR data sets, we show the potential of our designs. In particular, our designs scale to 256 GPUs with as low as 50 seconds of epoch times for ImageNet 1K datasets.