IKUN: Initialization to Keep snn training and generalization great with sUrrogate-stable variaNce
This addresses a specific bottleneck in SNN training for researchers and practitioners, offering a novel initialization method to enhance convergence and generalization, though it is incremental as it builds on existing surrogate gradient techniques.
The paper tackles the problem of weight initialization for spiking neural networks (SNNs), where traditional methods like Xavier and Kaiming are insufficient, by introducing IKUN, a variance-stabilizing initialization method integrated with surrogate gradient functions, which improves training efficiency by up to 50% and achieves 95% training accuracy and 91% generalization accuracy.
Weight initialization significantly impacts the convergence and performance of neural networks. While traditional methods like Xavier and Kaiming initialization are widely used, they often fall short for spiking neural networks (SNNs), which have distinct requirements compared to artificial neural networks (ANNs). To address this, we introduce \textbf{IKUN}, a variance-stabilizing initialization method integrated with surrogate gradient functions, specifically designed for SNNs. \textbf{IKUN} stabilizes signal propagation, accelerates convergence, and enhances generalization. Experiments show \textbf{IKUN} improves training efficiency by up to \textbf{50\%}, achieving \textbf{95\%} training accuracy and \textbf{91\%} generalization accuracy. Hessian analysis reveals that \textbf{IKUN}-trained models converge to flatter minima, characterized by Hessian eigenvalues near zero on the positive side, promoting better generalization. The method is open-sourced for further exploration: \href{https://github.com/MaeChd/SurrogateVarStabe}{https://github.com/MaeChd/SurrogateVarStabe}.