Gear Training: A new way to implement high-performance model-parallel training
This addresses the need for efficient distributed training in deep learning, but appears incremental as it builds on existing model-parallel concepts.
The paper tackles the problem of training deep neural networks in clusters by proposing a new model-parallel approach that splits the model and trains parts at different speeds, aiming for high performance.
The training of Deep Neural Networks usually needs tremendous computing resources. Therefore many deep models are trained in large cluster instead of single machine or GPU. Though major researchs at present try to run whole model on all machines by using asynchronous asynchronous stochastic gradient descent (ASGD), we present a new approach to train deep model parallely -- split the model and then seperately train different parts of it in different speed.