Exploring Temporal Information Dynamics in Spiking Neural Networks
This provides fundamental insights into SNN learning dynamics, which could help researchers design more efficient and robust neuromorphic systems, though it appears incremental in advancing understanding rather than achieving major performance breakthroughs.
The paper tackles the lack of explicit analysis of temporal information dynamics in Spiking Neural Networks (SNNs) by measuring Fisher Information of weights during training, finding that information concentrates in early timesteps, a phenomenon termed temporal information concentration. The result shows this concentration is crucial for robustness but has little effect on classification accuracy, and they propose an efficient iterative pruning method based on this observation.
Most existing Spiking Neural Network (SNN) works state that SNNs may utilize temporal information dynamics of spikes. However, an explicit analysis of temporal information dynamics is still missing. In this paper, we ask several important questions for providing a fundamental understanding of SNNs: What are temporal information dynamics inside SNNs? How can we measure the temporal information dynamics? How do the temporal information dynamics affect the overall learning performance? To answer these questions, we estimate the Fisher Information of the weights to measure the distribution of temporal information during training in an empirical manner. Surprisingly, as training goes on, Fisher information starts to concentrate in the early timesteps. After training, we observe that information becomes highly concentrated in earlier few timesteps, a phenomenon we refer to as temporal information concentration. We observe that the temporal information concentration phenomenon is a common learning feature of SNNs by conducting extensive experiments on various configurations such as architecture, dataset, optimization strategy, time constant, and timesteps. Furthermore, to reveal how temporal information concentration affects the performance of SNNs, we design a loss function to change the trend of temporal information. We find that temporal information concentration is crucial to building a robust SNN but has little effect on classification accuracy. Finally, we propose an efficient iterative pruning method based on our observation on temporal information concentration. Code is available at https://github.com/Intelligent-Computing-Lab-Yale/Exploring-Temporal-Information-Dynamics-in-Spiking-Neural-Networks.