High-performance deep spiking neural networks with 0.3 spikes per neuron
This work addresses the energy efficiency problem for neuromorphic computing by enabling high-performance SNNs with low spike rates, though it is incremental as it builds on existing mapping algorithms.
The paper tackled the challenge of training deep spiking neural networks (SNNs) by analyzing learning dynamics and identifying a vanishing-or-exploding gradient problem, achieving the same performance as artificial neural networks on image classification datasets like MNIST, CIFAR10, and PLACES365 with less than 0.3 spikes per neuron.
Communication by rare, binary spikes is a key factor for the energy efficiency of biological brains. However, it is harder to train biologically-inspired spiking neural networks (SNNs) than artificial neural networks (ANNs). This is puzzling given that theoretical results provide exact mapping algorithms from ANNs to SNNs with time-to-first-spike (TTFS) coding. In this paper we analyze in theory and simulation the learning dynamics of TTFS-networks and identify a specific instance of the vanishing-or-exploding gradient problem. While two choices of SNN mappings solve this problem at initialization, only the one with a constant slope of the neuron membrane potential at threshold guarantees the equivalence of the training trajectory between SNNs and ANNs with rectified linear units. We demonstrate that training deep SNN models achieves the exact same performance as that of ANNs, surpassing previous SNNs on image classification datasets such as MNIST/Fashion-MNIST, CIFAR10/CIFAR100 and PLACES365. Our SNN accomplishes high-performance classification with less than 0.3 spikes per neuron, lending itself for an energy-efficient implementation. We show that fine-tuning SNNs with our robust gradient descent algorithm enables their optimization for hardware implementations with low latency and resilience to noise and quantization.