SpikeTAD: Spiking Neural Networks for End-to-End Temporal Action Detection
This work addresses the need for low-power video understanding on mobile devices by introducing SNNs to temporal action detection, a task previously dominated by energy-intensive ANNs.
SpikeTAD is the first SNN-based end-to-end temporal action detection architecture, achieving 67.2% mAP on THUMOS14 and 37.42% on ActivityNet-1.3 while maintaining low power consumption.
Video understanding is a crucial part of computer vision, with numerous application scenarios. With the increasing popularity of mobile devices, an increasing number of efforts are trying to deploy video understanding models on them. However, existing video understanding models are difficult to deploy due to their large size and prohibitive power consumption. Spiking Neural Networks (SNNs) have shown bioplausibility and low power advantages over Artificial Neural Networks (ANNs), especially on neuromorphic chips which are regarded as essential components of future mobile devices. However, excessively long conversion time-steps and severe performance degradation problems limit their application. To solve the problems above, we explore the application of SNNs on temporal action detection (TAD), which is an important task in video understanding, and propose the first SNN-based end-to-end TAD architecture coined as SpikeTAD. While maintaining extremely low power consumption, SpikeTAD achieves an average mAP of 67.2% in THUMOS14 and 37.42% in ActivityNet-1.3, demonstrating the feasibility of a low-power TAD model. Our code is available at https://github.com/MCG-NJU/SpikeTAD.