Biologically inspired structure learning with reverse knowledge distillation for spiking neural networks
This work addresses the practical application gap in SNNs for sensory recognition tasks by introducing a dynamic structure learning approach, though it is incremental as it builds on existing pruning and knowledge distillation techniques.
The paper tackled the problem of redundancy in spiking neural networks (SNNs) by proposing an evolutionary-based structure construction method that integrates knowledge distillation and connection pruning, achieving competitive performance on CIFAR100 and DVS-Gesture datasets while reducing connection redundancy.
Spiking neural networks (SNNs) have superb characteristics in sensory information recognition tasks due to their biological plausibility. However, the performance of some current spiking-based models is limited by their structures which means either fully connected or too-deep structures bring too much redundancy. This redundancy from both connection and neurons is one of the key factors hindering the practical application of SNNs. Although Some pruning methods were proposed to tackle this problem, they normally ignored the fact the neural topology in the human brain could be adjusted dynamically. Inspired by this, this paper proposed an evolutionary-based structure construction method for constructing more reasonable SNNs. By integrating the knowledge distillation and connection pruning method, the synaptic connections in SNNs can be optimized dynamically to reach an optimal state. As a result, the structure of SNNs could not only absorb knowledge from the teacher model but also search for deep but sparse network topology. Experimental results on CIFAR100 and DVS-Gesture show that the proposed structure learning method can get pretty well performance while reducing the connection redundancy. The proposed method explores a novel dynamical way for structure learning from scratch in SNNs which could build a bridge to close the gap between deep learning and bio-inspired neural dynamics.