ReSpike: Residual Frames-based Hybrid Spiking Neural Networks for Efficient Action Recognition
This addresses the problem of inefficient action recognition for applications requiring low energy, though it is incremental as it builds on existing hybrid approaches.
The paper tackled the challenge of SNNs underperforming in video classification by proposing ReSpike, a hybrid ANN-SNN framework that decomposes videos into key and residual frames, achieving over 30% accuracy improvements on datasets like HMDB-51 and comparable performance to ANNs with better energy efficiency.
Spiking Neural Networks (SNNs) have emerged as a compelling, energy-efficient alternative to traditional Artificial Neural Networks (ANNs) for static image tasks such as image classification and segmentation. However, in the more complex video classification domain, SNN-based methods fall considerably short of ANN-based benchmarks due to the challenges in processing dense frame sequences. To bridge this gap, we propose ReSpike, a hybrid framework that synergizes the strengths of ANNs and SNNs to tackle action recognition tasks with high accuracy and low energy cost. By decomposing film clips into spatial and temporal components, i.e., RGB image Key Frames and event-like Residual Frames, ReSpike leverages ANN for learning spatial information and SNN for learning temporal information. In addition, we propose a multi-scale cross-attention mechanism for effective feature fusion. Compared to state-of-the-art SNN baselines, our ReSpike hybrid architecture demonstrates significant performance improvements (e.g., >30% absolute accuracy improvement on HMDB-51, UCF-101, and Kinetics-400). Furthermore, ReSpike achieves comparable performance with prior ANN approaches while bringing better accuracy-energy tradeoff.