TDFormer: A Top-Down Attention-Controlled Spiking Transformer
This addresses performance bottlenecks in spiking neural networks for researchers in neuromorphic computing, offering a novel architectural improvement.
The paper tackles the limitation of spiking neural networks in representing temporal information across time steps by introducing TDFormer, a model with a top-down feedback structure that improves mutual information and alleviates vanishing gradients, achieving state-of-the-art performance on ImageNet with 86.83% accuracy.
Traditional spiking neural networks (SNNs) can be viewed as a combination of multiple subnetworks with each running for one time step, where the parameters are shared, and the membrane potential serves as the only information link between them. However, the implicit nature of the membrane potential limits its ability to effectively represent temporal information. As a result, each time step cannot fully leverage information from previous time steps, seriously limiting the model's performance. Inspired by the top-down mechanism in the brain, we introduce TDFormer, a novel model with a top-down feedback structure that functions hierarchically and leverages high-order representations from earlier time steps to modulate the processing of low-order information at later stages. The feedback structure plays a role from two perspectives: 1) During forward propagation, our model increases the mutual information across time steps, indicating that richer temporal information is being transmitted and integrated in different time steps. 2) During backward propagation, we theoretically prove that the feedback structure alleviates the problem of vanishing gradients along the time dimension. We find that these mechanisms together significantly and consistently improve the model performance on multiple datasets. In particular, our model achieves state-of-the-art performance on ImageNet with an accuracy of 86.83%.