The fine line between dead neurons and sparsity in binarized spiking neural networks
This addresses the issue of sparse spike activity turning into no emission in spiking neural networks, which is an incremental improvement for efficient AI hardware.
The paper tackles the problem of dead neurons in binarized spiking neural networks by proposing threshold annealing as a warm-up method, achieving competitive results on four diverse datasets with binarized weights.
Spiking neural networks can compensate for quantization error by encoding information either in the temporal domain, or by processing discretized quantities in hidden states of higher precision. In theory, a wide dynamic range state-space enables multiple binarized inputs to be accumulated together, thus improving the representational capacity of individual neurons. This may be achieved by increasing the firing threshold, but make it too high and sparse spike activity turns into no spike emission. In this paper, we propose the use of `threshold annealing' as a warm-up method for firing thresholds. We show it enables the propagation of spikes across multiple layers where neurons would otherwise cease to fire, and in doing so, achieve highly competitive results on four diverse datasets, despite using binarized weights. Source code is available at https://github.com/jeshraghian/snn-tha/