Leapfrogging for parallelism in deep neural networks
arXiv:1801.04928v1
Originality Incremental advance
AI Analysis
This addresses the computational bottleneck of backpropagation for deep learning practitioners, though it appears incremental as it builds on existing parallelization methods.
The paper tackles the problem of parallelizing backpropagation in deep neural networks by introducing a technique called leapfrogging, which results in a savings of 1-1/k of a dominant term, where k is the number of threads or GPUs.
We present a technique, which we term leapfrogging, to parallelize back- propagation in deep neural networks. We show that this technique yields a savings of $1-1/k$ of a dominant term in backpropagation, where k is the number of threads (or gpus).