SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks
This enables faster training of sparse networks on commodity hardware, which is incremental as it builds on existing sparse training methods.
The paper tackles the problem of slow training for sparse neural networks by introducing SparseProp, an efficient sparse backpropagation algorithm that works with arbitrary sparsity and common layer types, achieving speedups in end-to-end runtime experiments on commodity CPUs.
We provide a new efficient version of the backpropagation algorithm, specialized to the case where the weights of the neural network being trained are sparse. Our algorithm is general, as it applies to arbitrary (unstructured) sparsity and common layer types (e.g., convolutional or linear). We provide a fast vectorized implementation on commodity CPUs, and show that it can yield speedups in end-to-end runtime experiments, both in transfer learning using already-sparsified networks, and in training sparse networks from scratch. Thus, our results provide the first support for sparse training on commodity hardware.