Probabilistic Modeling: Proving the Lottery Ticket Hypothesis in Spiking Neural Network
This work addresses the challenge of network pruning for SNNs, which are important for neuromorphic computing, by extending a foundational theory, though it is incremental in applying existing concepts to a new domain.
The authors tackled the problem of proving the Lottery Ticket Hypothesis (LTH) in Spiking Neural Networks (SNNs), which was previously limited to continuous activation functions, and they demonstrated through theoretical and experimental results that LTH holds in SNNs, leading to a new pruning criterion that outperforms baseline methods.
The Lottery Ticket Hypothesis (LTH) states that a randomly-initialized large neural network contains a small sub-network (i.e., winning tickets) which, when trained in isolation, can achieve comparable performance to the large network. LTH opens up a new path for network pruning. Existing proofs of LTH in Artificial Neural Networks (ANNs) are based on continuous activation functions, such as ReLU, which satisfying the Lipschitz condition. However, these theoretical methods are not applicable in Spiking Neural Networks (SNNs) due to the discontinuous of spiking function. We argue that it is possible to extend the scope of LTH by eliminating Lipschitz condition. Specifically, we propose a novel probabilistic modeling approach for spiking neurons with complicated spatio-temporal dynamics. Then we theoretically and experimentally prove that LTH holds in SNNs. According to our theorem, we conclude that pruning directly in accordance with the weight size in existing SNNs is clearly not optimal. We further design a new criterion for pruning based on our theory, which achieves better pruning results than baseline.