Advancing Direct Training for Spiking Neural Networks with Circulate-Firing Neurons and Learnable Gradients
This work addresses the performance gap between SNNs and ANNs by enhancing information representation and gradient propagation, benefiting the SNN community seeking high-performance models.
The paper proposes a direct training algorithm for Spiking Neural Networks (SNNs) that introduces circulate-firing neurons, learnable surrogate gradients, and a balanced loss function, achieving competitive performance across multiple datasets and outperforming existing methods on Transformer architectures.
Spiking Neural Networks (SNNs) have emerged with promising energy-efficient property, yet a substantial performance gap persists compared to Artificial Neural Networks (ANNs). This gap stems from at least two key limitations: first, conventional spiking neurons offer limited information representation capacity, underutilizing the rich dynamics of membrane potentials; second, fixed surrogate gradient (SG) functions across time steps leads to imprecise gradient propagation, impeding effective direct training. To address these two challenges, we propose a new direct training algorithm with three core innovations: first, a circulate-firing spiking neuron model that enhances information representation capacity by leveraging membrane potentials more effectively; second, a time-step-wise learnable surrogate gradient function, enabling accurate gradient estimation during backpropagation; third, a positive-negative balanced loss function to achieve equilibrium between positive and negative membrane potentials and further boost SNN performance. Extensive experiments demonstrate that our methods achieve competitive performance across multiple datasets. Our methods can generalize seamlessly to advanced architectures of Transformer, consistently outperforming existing methods. Our work highlights the effectiveness of further harnessing intrinsic membrane dynamics of SNNs for performance improvement, and thus open a new avenue for advancing high-performance spiking neural architectures.