Spiking and Event-driven Neuromorphic Mamba Models for Efficient Speech Recognition
For researchers and engineers deploying ASR on edge devices, this work demonstrates that neuromorphic approaches can achieve high sparsity with minimal accuracy loss, but the gains are incremental as they apply existing neuromorphic techniques to a specific model.
The paper introduces event-driven and spiking variants of the SpeechMamba model for ASR, achieving over 60% activation sparsity with less than 1% accuracy degradation on LibriSpeech, and over 70% sparsity with 30% fewer parameters. A cycle-accurate simulator enables algorithm-hardware co-exploration, yielding over 10% additional efficiency improvements.
Deep learning has greatly advanced automatic speech recognition (ASR), enabling widespread deployment on edge devices such as smartphones and smart home systems. However, the computational and energy demands of deep neural networks pose significant challenges for such resource-constrained deployments, introducing latency and limiting real-time interaction. Neuromorphic computing offers a promising solution by introducing activation sparsity through spiking neural networks (SNNs) and event-driven neural networks, converting dense operations into sparse computations. However, a study that evaluates the hardware benefits of different neuromorphic strategies remains lacking for ASR. This paper explores spiking and event-driven neuromorphic neural networks to improve activation sparsity in the state-of-the-art SpeechMamba model for ASR. We introduce an event-driven SpeechMamba with FATReLU activation, achieving over 60% activation sparsity with less than 1% accuracy degradation on LibriSpeech. We also propose a spiking SpeechMamba that attains over 70% sparsity while using 30% fewer parameters than comparable SNNs. Finally, we develop a cycle-accurate event-driven simulator enabling flexible algorithm-hardware co-exploration, which helps us identify computational bottlenecks and yields over 10% additional efficiency improvements.