Training-Free ANN-to-SNN Conversion for High-Performance Spiking Transformer
This provides an efficient and scalable method for deploying energy-efficient Spiking Transformers in real-world applications, though it is incremental as it builds on existing ANN-to-SNN conversion approaches.
The paper tackles the problem of converting pre-trained Artificial Neural Networks (ANNs) to Spiking Neural Networks (SNNs) for Transformers without training, achieving near-lossless accuracy with significantly lower latency across tasks like computer vision and natural language processing.
Leveraging the event-driven paradigm, Spiking Neural Networks (SNNs) offer a promising approach for constructing energy-efficient Transformer architectures. Compared to directly trained Spiking Transformers, ANN-to-SNN conversion methods bypass the high training costs. However, existing methods still suffer from notable limitations, failing to effectively handle nonlinear operations in Transformer architectures and requiring additional fine-tuning processes for pre-trained ANNs. To address these issues, we propose a high-performance and training-free ANN-to-SNN conversion framework tailored for Transformer architectures. Specifically, we introduce a Multi-basis Exponential Decay (MBE) neuron, which employs an exponential decay strategy and multi-basis encoding method to efficiently approximate various nonlinear operations. It removes the requirement for weight modifications in pre-trained ANNs. Extensive experiments across diverse tasks (CV, NLU, NLG) and mainstream Transformer architectures (ViT, RoBERTa, GPT-2) demonstrate that our method achieves near-lossless conversion accuracy with significantly lower latency. This provides a promising pathway for the efficient and scalable deployment of Spiking Transformers in real-world applications.