MXNET-MPI: Embedding MPI parallelism in Parameter Server Task Model for scaling Deep Learning
This work addresses scaling challenges in deep learning for researchers and practitioners by combining existing paradigms, though it is incremental as it builds upon prior PS and MPI approaches.
The paper tackles the problem of scaling deep learning by proposing a framework that integrates MPI parallelism into the Parameter Server model, enabling novel communication-avoiding algorithms like Elastic SGD to improve convergence rates, with empirical validation on large-scale ImageNet 1K data.
Existing Deep Learning frameworks exclusively use either Parameter Server(PS) approach or MPI parallelism. In this paper, we discuss the drawbacks of such approaches and propose a generic framework supporting both PS and MPI programming paradigms, co-existing at the same time. The key advantage of the new model is to embed the scaling benefits of MPI parallelism into the loosely coupled PS task model. Apart from providing a practical usage model of MPI in cloud, such framework allows for novel communication avoiding algorithms that do parameter averaging in Stochastic Gradient Descent(SGD) approaches. We show how MPI and PS models can synergestically apply algorithms such as Elastic SGD to improve the rate of convergence against existing approaches. These new algorithms directly help scaling SGD clusterwide. Further, we also optimize the critical component of the framework, namely global aggregation or allreduce using a novel concept of tensor collectives. These treat a group of vectors on a node as a single object allowing for the existing single vector algorithms to be directly applicable. We back our claims with sufficient emperical evidence using large scale ImageNet 1K data. Our framework is built upon MXNET but the design is generic and can be adapted to other popular DL infrastructures.