A Data and Model-Parallel, Distributed and Scalable Framework for Training of Deep Networks in Apache Spark
This framework enables scalable and economical training of large deep networks on commodity CPU clusters, addressing a bottleneck for researchers and practitioners with limited access to GPUs or supercomputers.
The authors tackled the problem of expensive and time-consuming deep network training by developing a distributed framework in Apache Spark that supports both data and model parallelism, achieving up to 11x speedup for CNNs while maintaining accuracy.
Training deep networks is expensive and time-consuming with the training period increasing with data size and growth in model parameters. In this paper, we provide a framework for distributed training of deep networks over a cluster of CPUs in Apache Spark. The framework implements both Data Parallelism and Model Parallelism making it suitable to use for deep networks which require huge training data and model parameters which are too big to fit into the memory of a single machine. It can be scaled easily over a cluster of cheap commodity hardware to attain significant speedup and obtain better results making it quite economical as compared to farm of GPUs and supercomputers. We have proposed a new algorithm for training of deep networks for the case when the network is partitioned across the machines (Model Parallelism) along with detailed cost analysis and proof of convergence of the same. We have developed implementations for Fully-Connected Feedforward Networks, Convolutional Neural Networks, Recurrent Neural Networks and Long Short-Term Memory architectures. We present the results of extensive simulations demonstrating the speedup and accuracy obtained by our framework for different sizes of the data and model parameters with variation in the number of worker cores/partitions; thereby showing that our proposed framework can achieve significant speedup (upto 11X for CNN) and is also quite scalable.