Training Deep Spiking Neural Networks
This work addresses the problem of inefficient training for deep SNNs, which is crucial for enabling energy-efficient neuromorphic computing, though it is incremental as it builds on existing methods like surrogate gradients and batch normalization.
The paper tackled the challenge of training deep spiking neural networks (SNNs) by addressing gradient issues and applying batch normalization, enabling training of ResNet50-like SNNs on datasets like CIFAR100 and Imagenette. The result was SNNs that achieve good accuracy with as low as 10 inference time steps, significantly fewer than the 1000 steps required by converted SNNs, though they still lag behind analogous ANNs in accuracy.
Computation using brain-inspired spiking neural networks (SNNs) with neuromorphic hardware may offer orders of magnitude higher energy efficiency compared to the current analog neural networks (ANNs). Unfortunately, training SNNs with the same number of layers as state of the art ANNs remains a challenge. To our knowledge the only method which is successful in this regard is supervised training of ANN and then converting it to SNN. In this work we directly train deep SNNs using backpropagation with surrogate gradient and find that due to implicitly recurrent nature of feed forward SNN's the exploding or vanishing gradient problem severely hinders their training. We show that this problem can be solved by tuning the surrogate gradient function. We also propose using batch normalization from ANN literature on input currents of SNN neurons. Using these improvements we show that is is possible to train SNN with ResNet50 architecture on CIFAR100 and Imagenette object recognition datasets. The trained SNN falls behind in accuracy compared to analogous ANN but requires several orders of magnitude less inference time steps (as low as 10) to reach good accuracy compared to SNNs obtained by conversion from ANN which require on the order of 1000 time steps.