Towards Understanding the Effect of Leak in Spiking Neural Networks
This addresses the trade-offs in neuron model design for SNNs, offering insights for more efficient and robust neuromorphic computing, though it is incremental in analyzing existing models.
The paper investigates the computational impact of leak in spiking neural networks, finding that leaky models improve robustness and generalization but reduce sparsity, with frequency analysis showing leak filters high-frequency noise.
Spiking Neural Networks (SNNs) are being explored to emulate the astounding capabilities of human brain that can learn and compute functions robustly and efficiently with noisy spiking activities. A variety of spiking neuron models have been proposed to resemble biological neuronal functionalities. With varying levels of bio-fidelity, these models often contain a leak path in their internal states, called membrane potentials. While the leaky models have been argued as more bioplausible, a comparative analysis between models with and without leak from a purely computational point of view demands attention. In this paper, we investigate the questions regarding the justification of leak and the pros and cons of using leaky behavior. Our experimental results reveal that leaky neuron model provides improved robustness and better generalization compared to models with no leak. However, leak decreases the sparsity of computation contrary to the common notion. Through a frequency domain analysis, we demonstrate the effect of leak in eliminating the high-frequency components from the input, thus enabling SNNs to be more robust against noisy spike-inputs.