Temporally Efficient Deep Learning with Spikes
This addresses energy waste in deep learning for processing temporal data like video or audio, though it is incremental as it builds on existing backpropagation methods.
The paper tackles the problem of computational inefficiency in deep learning due to temporal redundancy in sensory data by introducing a backpropagation variant where computation scales with data change rate, achieving performance comparable to a Multilayer Perceptron on MNIST and its temporal variant while communicating only discrete values.
The vast majority of natural sensory data is temporally redundant. Video frames or audio samples which are sampled at nearby points in time tend to have similar values. Typically, deep learning algorithms take no advantage of this redundancy to reduce computation. This can be an obscene waste of energy. We present a variant on backpropagation for neural networks in which computation scales with the rate of change of the data - not the rate at which we process the data. We do this by having neurons communicate a combination of their state, and their temporal change in state. Intriguingly, this simple communication rule give rise to units that resemble biologically-inspired leaky integrate-and-fire neurons, and to a weight-update rule that is equivalent to a form of Spike-Timing Dependent Plasticity (STDP), a synaptic learning rule observed in the brain. We demonstrate that on MNIST and a temporal variant of MNIST, our algorithm performs about as well as a Multilayer Perceptron trained with backpropagation, despite only communicating discrete values between layers.