A Survey and Empirical Evaluation of Parallel Deep Learning Frameworks
This survey helps researchers and practitioners choose efficient frameworks for large-scale neural network training, but it is incremental as it reviews and benchmarks existing methods.
The paper compares state-of-the-art frameworks for distributed deep learning, identifying parallelism types and evaluating their performance on large image and language tasks, with results showing differences in statistical efficiency and memory consumption.
The field of deep learning has witnessed a remarkable shift towards extremely compute- and memory-intensive neural networks. These newer larger models have enabled researchers to advance state-of-the-art tools across a variety of fields. This phenomenon has spurred the development of algorithms for distributed training of neural networks over a larger number of hardware accelerators. In this paper, we discuss and compare current state-of-the-art frameworks for large scale distributed deep learning. First, we survey current practices in distributed learning and identify the different types of parallelism used. Then, we present empirical results comparing their performance on large image and language training tasks. Additionally, we address their statistical efficiency and memory consumption behavior. Based on our results, we discuss algorithmic and implementation portions of each framework which hinder performance.