Temporal Surrogate Back-propagation for Spiking Neural Networks
This work provides a theoretical completion for SNN training algorithms by accounting for temporal dependencies in the reset mechanism, which is relevant for researchers developing more biologically plausible and energy-efficient neural networks.
This paper addresses the non-differentiability of spike behavior in Spiking Neural Networks (SNNs) that prevents direct application of back-propagation. The authors investigate a previously omitted temporal dependency introduced by the reset mechanism between steps, finding that while it improves robustness to learning-rate adjustments on a toy dataset, it offers little improvement on larger tasks like CIFAR-10.
Spiking neural networks (SNN) are usually more energy-efficient as compared to Artificial neural networks (ANN), and the way they work has a great similarity with our brain. Back-propagation (BP) has shown its strong power in training ANN in recent years. However, since spike behavior is non-differentiable, BP cannot be applied to SNN directly. Although prior works demonstrated several ways to approximate the BP-gradient in both spatial and temporal directions either through surrogate gradient or randomness, they omitted the temporal dependency introduced by the reset mechanism between each step. In this article, we target on theoretical completion and investigate the effect of the missing term thoroughly. By adding the temporal dependency of the reset mechanism, the new algorithm is more robust to learning-rate adjustments on a toy dataset but does not show much improvement on larger learning tasks like CIFAR-10. Empirically speaking, the benefits of the missing term are not worth the additional computational overhead. In many cases, the missing term can be ignored.