DTA: Dual Temporal-channel-wise Attention for Spiking Neural Networks
This is an incremental improvement for SNNs, focusing on better attention mechanisms to boost energy-efficient AI models.
The paper tackles the problem of enhancing temporal information utilization in Spiking Neural Networks (SNNs) by proposing a Dual Temporal-channel-wise Attention (DTA) mechanism, which achieves state-of-the-art performance on datasets like CIFAR10, CIFAR100, ImageNet-1k, and CIFAR10-DVS.
Spiking Neural Networks (SNNs) present a more energy-efficient alternative to Artificial Neural Networks (ANNs) by harnessing spatio-temporal dynamics and event-driven spikes. Effective utilization of temporal information is crucial for SNNs, leading to the exploration of attention mechanisms to enhance this capability. Conventional attention operations either apply identical operation or employ non-identical operations across target dimensions. We identify that these approaches provide distinct perspectives on temporal information. To leverage the strengths of both operations, we propose a novel Dual Temporal-channel-wise Attention (DTA) mechanism that integrates both identical/non-identical attention strategies. To the best of our knowledge, this is the first attempt to concentrate on both the correlation and dependency of temporal-channel using both identical and non-identical attention operations. Experimental results demonstrate that the DTA mechanism achieves state-of-the-art performance on both static datasets (CIFAR10, CIFAR100, ImageNet-1k) and dynamic dataset (CIFAR10-DVS), elevating spike representation and capturing complex temporal-channel relationship. We open-source our code: https://github.com/MnJnKIM/DTA-SNN.