AutoST: Training-free Neural Architecture Search for Spiking Transformers
This addresses the architectural gap in Spiking Transformers for researchers and practitioners in neuromorphic computing, offering an incremental improvement over existing training-free NAS methods.
The paper tackles the suboptimal performance of Spiking Transformers by introducing AutoST, a training-free neural architecture search method that uses FLOPs as a metric to rapidly identify high-performance architectures, achieving state-of-the-art results on static and neuromorphic datasets.
Spiking Transformers have gained considerable attention because they achieve both the energy efficiency of Spiking Neural Networks (SNNs) and the high capacity of Transformers. However, the existing Spiking Transformer architectures, derived from Artificial Neural Networks (ANNs), exhibit a notable architectural gap, resulting in suboptimal performance compared to their ANN counterparts. Manually discovering optimal architectures is time-consuming. To address these limitations, we introduce AutoST, a training-free NAS method for Spiking Transformers, to rapidly identify high-performance Spiking Transformer architectures. Unlike existing training-free NAS methods, which struggle with the non-differentiability and high sparsity inherent in SNNs, we propose to utilize Floating-Point Operations (FLOPs) as a performance metric, which is independent of model computations and training dynamics, leading to a stronger correlation with performance. Our extensive experiments show that AutoST models outperform state-of-the-art manually or automatically designed SNN architectures on static and neuromorphic datasets. Full code, model, and data are released for reproduction.