Differentiable architecture search with multi-dimensional attention for spiking neural networks
This addresses the problem of inefficient SNN structures for researchers in low-power AI, though it is incremental as it builds on existing NAS and attention methods.
The paper tackled the sub-optimal performance of Spiking Neural Networks (SNNs) by proposing MA-DARTS, a differentiable architecture search method with multi-dimensional attention, achieving state-of-the-art accuracies of 94.40% on CIFAR10 and 76.52% on CIFAR100.
Spiking Neural Networks (SNNs) have gained enormous popularity in the field of artificial intelligence due to their low power consumption. However, the majority of SNN methods directly inherit the structure of Artificial Neural Networks (ANN), usually leading to sub-optimal model performance in SNNs. To alleviate this problem, we integrate Neural Architecture Search (NAS) method and propose Multi-Attention Differentiable Architecture Search (MA-DARTS) to directly automate the search for the optimal network structure of SNNs. Initially, we defined a differentiable two-level search space and conducted experiments within micro architecture under a fixed layer. Then, we incorporated a multi-dimensional attention mechanism and implemented the MA-DARTS algorithm in this search space. Comprehensive experiments demonstrate our model achieves state-of-the-art performance on classification compared to other methods under the same parameters with 94.40% accuracy on CIFAR10 dataset and 76.52% accuracy on CIFAR100 dataset. Additionally, we monitored and assessed the number of spikes (NoS) in each cell during the whole experiment. Notably, the number of spikes of the whole model stabilized at approximately 110K in validation and 100k in training on datasets.